SparkContext

final def !=(arg0: AnyRef): Boolean

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: AnyRef): Boolean

final def ==(arg0: Any): Boolean

def accumulable[T, R](initialValue: T)(implicit param: AccumulableParam[T, R]): Accumulable[T, R]

Create an org.apache.spark.Accumulable shared variable, to which tasks can add values with +=.

def accumulableCollection[R, T](initialValue: R)(implicit arg0: (R) ⇒ Growable[T] with TraversableOnce[T] with Serializable): Accumulable[R, T]

Create an accumulator from a "mutable collection" type.

def accumulator[T](initialValue: T)(implicit param: AccumulatorParam[T]): Accumulator[T]

Create an org.apache.spark.Accumulator variable of a given type, which tasks can "add" values to using the += method.

def addFile(path: String): Unit

Add a file to be downloaded with this Spark job on every node.

def addJar(path: String): Unit

Adds a JAR dependency for all tasks to be executed on this SparkContext in the future.

def addSparkListener(listener: SparkListener): Unit

val appName: String

final def asInstanceOf[T0]: T0

def broadcast[T](value: T): Broadcast[T]

Broadcast a read-only variable to the cluster, returning a org.apache.spark.broadcast.Broadcast object for reading it in distributed functions.

def cancelAllJobs(): Unit

Cancel all jobs that have been scheduled or are running.

def cancelJobGroup(groupId: String): Unit

Cancel active jobs for the specified group.

def checkpointFile[T](path: String)(implicit arg0: ClassTag[T]): RDD[T]

def clearCallSite(): Unit

Support function for API backtraces.

def clearFiles(): Unit

Clear the job's list of files added by addFile so that they do not get downloaded to any new nodes.

def clearJars(): Unit

Clear the job's list of JARs added by addJar so that they do not get downloaded to any new nodes.

def clearJobGroup(): Unit

Clear the current thread's job group ID and its description.

def clone(): AnyRef

def defaultMinSplits: Int

Default min number of partitions for Hadoop RDDs when not given by user

def defaultParallelism: Int

Default level of parallelism to use when not given by user (e.

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

def getAllPools: ArrayBuffer[Schedulable]

Return pools for fair scheduler TODO(xiajunluan): We should take nested pools into account

def getCheckpointDir: Option[String]

final def getClass(): Class[_]

def getConf: SparkConf

Return a copy of this SparkContext's configuration.

def getExecutorMemoryStatus: Map[String, (Long, Long)]

Return a map from the slave to the max memory available for caching and the remaining memory available for caching.

def getExecutorStorageStatus: Array[StorageStatus]

Return information about blocks stored in all of the slaves

def getLocalProperty(key: String): String

Get a local property set in this thread, or null if it is missing.

def getPersistentRDDs: Map[Int, RDD[_]]

Returns an immutable map of RDDs that have marked themselves as persistent via cache() call.

def getPoolForName(pool: String): Option[Schedulable]

Return the pool associated with the given name, if one exists

def getRDDStorageInfo: Array[RDDInfo]

Return information about what RDDs are cached, if they are in mem or on disk, how much space they take, etc.

def getSchedulingMode: SchedulingMode

Return current scheduling mode

def getStageInfo: Map[Stage, StageInfo]

val hadoopConfiguration: Configuration

A default Hadoop Configuration for the Hadoop code (e.

def hadoopFile[K, V, F <: InputFormat[K, V]](path: String)(implicit km: ClassTag[K], vm: ClassTag[V], fm: ClassTag[F]): RDD[(K, V)]

Smarter version of hadoopFile() that uses class tags to figure out the classes of keys, values and the InputFormat so that users don't need to pass them directly.

val file = sparkContext.hadoopFile[LongWritable, Text, TextInputFormat](path)

def hadoopFile[K, V, F <: InputFormat[K, V]](path: String, minSplits: Int)(implicit km: ClassTag[K], vm: ClassTag[V], fm: ClassTag[F]): RDD[(K, V)]

Smarter version of hadoopFile() that uses class tags to figure out the classes of keys, values and the InputFormat so that users don't need to pass them directly.

val file = sparkContext.hadoopFile[LongWritable, Text, TextInputFormat](path, minSplits)

def hadoopFile[K, V](path: String, inputFormatClass: Class[_ <: InputFormat[K, V]], keyClass: Class[K], valueClass: Class[V], minSplits: Int = defaultMinSplits): RDD[(K, V)]

Get an RDD for a Hadoop file with an arbitrary InputFormat

def hadoopRDD[K, V](conf: JobConf, inputFormatClass: Class[_ <: InputFormat[K, V]], keyClass: Class[K], valueClass: Class[V], minSplits: Int = defaultMinSplits): RDD[(K, V)]

Get an RDD for a Hadoop-readable dataset from a Hadoop JobConf given its InputFormat and other necessary info (e.

def hashCode(): Int

def initDriverMetrics(): Unit

def initLocalProperties(): Unit

final def isInstanceOf[T0]: Boolean

val isLocal: Boolean

def isTraceEnabled(): Boolean

val jars: Seq[String]

def log: Logger

def logDebug(msg: ⇒ String, throwable: Throwable): Unit

def logDebug(msg: ⇒ String): Unit

def logError(msg: ⇒ String, throwable: Throwable): Unit

def logError(msg: ⇒ String): Unit

def logInfo(msg: ⇒ String, throwable: Throwable): Unit

def logInfo(msg: ⇒ String): Unit

def logTrace(msg: ⇒ String, throwable: Throwable): Unit

def logTrace(msg: ⇒ String): Unit

def logWarning(msg: ⇒ String, throwable: Throwable): Unit

def logWarning(msg: ⇒ String): Unit

def makeRDD[T](seq: Seq[(T, Seq[String])])(implicit arg0: ClassTag[T]): RDD[T]

Distribute a local Scala collection to form an RDD, with one or more location preferences (hostnames of Spark nodes) for each object.

def makeRDD[T](seq: Seq[T], numSlices: Int = defaultParallelism)(implicit arg0: ClassTag[T]): RDD[T]

Distribute a local Scala collection to form an RDD.

val master: String

final def ne(arg0: AnyRef): Boolean

def newAPIHadoopFile[K, V, F <: InputFormat[K, V]](path: String, fClass: Class[F], kClass: Class[K], vClass: Class[V], conf: Configuration = hadoopConfiguration): RDD[(K, V)]

Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.

def newAPIHadoopFile[K, V, F <: InputFormat[K, V]](path: String)(implicit km: ClassTag[K], vm: ClassTag[V], fm: ClassTag[F]): RDD[(K, V)]

Get an RDD for a Hadoop file with an arbitrary new API InputFormat.

def newAPIHadoopRDD[K, V, F <: InputFormat[K, V]](conf: Configuration = hadoopConfiguration, fClass: Class[F], kClass: Class[K], vClass: Class[V]): RDD[(K, V)]

Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.

final def notify(): Unit

final def notifyAll(): Unit

def objectFile[T](path: String, minSplits: Int = defaultMinSplits)(implicit arg0: ClassTag[T]): RDD[T]

Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition.

def parallelize[T](seq: Seq[T], numSlices: Int = defaultParallelism)(implicit arg0: ClassTag[T]): RDD[T]

Distribute a local Scala collection to form an RDD.

val preferredNodeLocationData: Map[String, Set[SplitInfo]]

used in YARN mode to select nodes to launch containers on.

def runApproximateJob[T, U, R](rdd: RDD[T], func: (TaskContext, Iterator[T]) ⇒ U, evaluator: ApproximateEvaluator[U, R], timeout: Long): PartialResult[R]

Run a job that can return approximate results.

def runJob[T, U](rdd: RDD[T], processPartition: (Iterator[T]) ⇒ U, resultHandler: (Int, U) ⇒ Unit)(implicit arg0: ClassTag[U]): Unit

Run a job on all partitions in an RDD and pass the results to a handler function.

def runJob[T, U](rdd: RDD[T], processPartition: (TaskContext, Iterator[T]) ⇒ U, resultHandler: (Int, U) ⇒ Unit)(implicit arg0: ClassTag[U]): Unit

Run a job on all partitions in an RDD and pass the results to a handler function.

def runJob[T, U](rdd: RDD[T], func: (Iterator[T]) ⇒ U)(implicit arg0: ClassTag[U]): Array[U]

Run a job on all partitions in an RDD and return the results in an array.

def runJob[T, U](rdd: RDD[T], func: (TaskContext, Iterator[T]) ⇒ U)(implicit arg0: ClassTag[U]): Array[U]

Run a job on all partitions in an RDD and return the results in an array.

def runJob[T, U](rdd: RDD[T], func: (Iterator[T]) ⇒ U, partitions: Seq[Int], allowLocal: Boolean)(implicit arg0: ClassTag[U]): Array[U]

Run a job on a given set of partitions of an RDD, but take a function of type Iterator[T] => U instead of (TaskContext, Iterator[T]) => U.

def runJob[T, U](rdd: RDD[T], func: (TaskContext, Iterator[T]) ⇒ U, partitions: Seq[Int], allowLocal: Boolean)(implicit arg0: ClassTag[U]): Array[U]

Run a function on a given set of partitions in an RDD and return the results as an array.

def runJob[T, U](rdd: RDD[T], func: (TaskContext, Iterator[T]) ⇒ U, partitions: Seq[Int], allowLocal: Boolean, resultHandler: (Int, U) ⇒ Unit)(implicit arg0: ClassTag[U]): Unit

Run a function on a given set of partitions in an RDD and pass the results to the given handler function.

def sequenceFile[K, V](path: String, minSplits: Int = defaultMinSplits)(implicit km: ClassTag[K], vm: ClassTag[V], kcf: () ⇒ WritableConverter[K], vcf: () ⇒ WritableConverter[V]): RDD[(K, V)]

Version of sequenceFile() for types implicitly convertible to Writables through a WritableConverter.

sparkContext.sequenceFile[String, Int](path, ...)

def sequenceFile[K, V](path: String, keyClass: Class[K], valueClass: Class[V]): RDD[(K, V)]

Get an RDD for a Hadoop SequenceFile with given key and value types.

def sequenceFile[K, V](path: String, keyClass: Class[K], valueClass: Class[V], minSplits: Int): RDD[(K, V)]

Get an RDD for a Hadoop SequenceFile with given key and value types.

def setCallSite(site: String): Unit

Support function for API backtraces.

def setCheckpointDir(directory: String): Unit

Set the directory under which RDDs are going to be checkpointed.

def setJobGroup(groupId: String, description: String): Unit

Assigns a group ID to all the jobs started by this thread until the group ID is set to a different value or cleared.

// In the main thread:
sc.setJobGroup("some_job_to_cancel", "some job description")
sc.parallelize(1 to 10000, 2).map { i => Thread.sleep(10); i }.count()

// In a separate thread:
sc.cancelJobGroup("some_job_to_cancel")

def setLocalProperty(key: String, value: String): Unit

Set a local property that affects jobs submitted from this thread, such as the Spark fair scheduler pool.

val sparkUser: String

val startTime: Long

def stop(): Unit

Shut down the SparkContext.

def submitJob[T, U, R](rdd: RDD[T], processPartition: (Iterator[T]) ⇒ U, partitions: Seq[Int], resultHandler: (Int, U) ⇒ Unit, resultFunc: ⇒ R): SimpleFutureAction[R]

Submit a job for execution and return a FutureJob holding the result.

final def synchronized[T0](arg0: ⇒ T0): T0

def textFile(path: String, minSplits: Int = defaultMinSplits): RDD[String]

Read a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings.

def toString(): String

def union[T](first: RDD[T], rest: RDD[T]*)(implicit arg0: ClassTag[T]): RDD[T]

Build the union of a list of RDDs passed as variable-length arguments.

def union[T](rdds: Seq[RDD[T]])(implicit arg0: ClassTag[T]): RDD[T]

Build the union of a list of RDDs.

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

def setJobDescription(value: String): Unit

Set a human readable description of the current job.