org.apache.spark.internal.io

Class SparkHadoopWriter

Object

org.apache.spark.internal.io.SparkHadoopWriter

public class SparkHadoopWriter
extends Object

A helper object that saves an RDD using a Hadoop OutputFormat.

Constructor Summary

Constructors

Constructor and Description
SparkHadoopWriter()

Method Summary

Modifier and Type	Method and Description
static <K,V> void	write(RDD<scala.Tuple2<K,V>> rdd, HadoopWriteConfigUtil<K,V> config, scala.reflect.ClassTag<V> evidence$1) Basic work flow of this command is: 1.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkHadoopWriter

public SparkHadoopWriter()

Method Detail

write

public static <K,V> void write(RDD<scala.Tuple2<K,V>> rdd,
                               HadoopWriteConfigUtil<K,V> config,
                               scala.reflect.ClassTag<V> evidence$1)

Basic work flow of this command is: 1. Driver side setup, prepare the data source and hadoop configuration for the write job to be issued. 2. Issues a write job consists of one or more executor side tasks, each of which writes all rows within an RDD partition. 3. If no exception is thrown in a task, commits that task, otherwise aborts that task; If any exception is thrown during task commitment, also aborts that task. 4. If all tasks are committed, commit the job, otherwise aborts the job; If any exception is thrown during job commitment, also aborts the job.

Parameters:

rdd - (undocumented)

config - (undocumented)

evidence$1 - (undocumented)