SparkContext

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

def addFile(path: String, recursive: Boolean): Unit

Add a file to be downloaded with this Spark job on every node.

def addFile(path: String): Unit

Add a file to be downloaded with this Spark job on every node.

def addJar(path: String): Unit

Adds a JAR dependency for all tasks to be executed on this SparkContext in the future.

def addSparkListener(listener: SparkListenerInterface): Unit

:: DeveloperApi :: Register a listener to receive up-calls from events that happen during execution.

def appName: String

def applicationAttemptId: Option[String]

def applicationId: String

A unique identifier for the Spark application.

final def asInstanceOf[T0]: T0

def binaryFiles(path: String, minPartitions: Int = defaultMinPartitions): RDD[(String, PortableDataStream)]

Get an RDD for a Hadoop-readable dataset as PortableDataStream for each file (useful for binary data)

hdfs://a-hdfs-path/part-00000
hdfs://a-hdfs-path/part-00001
...
hdfs://a-hdfs-path/part-nnnnn

(a-hdfs-path/part-00000, its content)
(a-hdfs-path/part-00001, its content)
...
(a-hdfs-path/part-nnnnn, its content)

def binaryRecords(path: String, recordLength: Int, conf: Configuration = hadoopConfiguration): RDD[Array[Byte]]

Load data from a flat binary file, assuming the length of each record is constant.

def broadcast[T](value: T)(implicit arg0: ClassTag[T]): Broadcast[T]

Broadcast a read-only variable to the cluster, returning a org.apache.spark.broadcast.Broadcast object for reading it in distributed functions.

def cancelAllJobs(): Unit

Cancel all jobs that have been scheduled or are running.

def cancelJobGroup(groupId: String): Unit

Cancel active jobs for the specified group.

def checkpointFile[T](path: String)(implicit arg0: ClassTag[T]): RDD[T]

def clearCallSite(): Unit

Clear the thread-local property for overriding the call sites of actions and RDDs.

def clearJobGroup(): Unit

Clear the current thread's job group ID and its description.

def clone(): AnyRef

def collectionAccumulator[T](name: String): CollectionAccumulator[T]

Create and register a CollectionAccumulator, which starts with empty list and accumulates inputs by adding them into the list.

def collectionAccumulator[T]: CollectionAccumulator[T]

Create and register a CollectionAccumulator, which starts with empty list and accumulates inputs by adding them into the list.

def defaultMinPartitions: Int

Default min number of partitions for Hadoop RDDs when not given by user Notice that we use math.min so the "defaultMinPartitions" cannot be higher than 2.

def defaultParallelism: Int

Default level of parallelism to use when not given by user (e.g.

def deployMode: String

def doubleAccumulator(name: String): DoubleAccumulator

Create and register a double accumulator, which starts with 0 and accumulates inputs by add.

def doubleAccumulator: DoubleAccumulator

Create and register a double accumulator, which starts with 0 and accumulates inputs by add.

def emptyRDD[T](implicit arg0: ClassTag[T]): RDD[T]

Get an RDD that has no partitions or elements.

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def files: Seq[String]

def finalize(): Unit

def getAllPools: Seq[Schedulable]

:: DeveloperApi :: Return pools for fair scheduler

def getCheckpointDir: Option[String]

final def getClass(): Class[_]

def getConf: SparkConf

Return a copy of this SparkContext's configuration.

def getExecutorMemoryStatus: Map[String, (Long, Long)]

Return a map from the slave to the max memory available for caching and the remaining memory available for caching.

def getExecutorStorageStatus: Array[StorageStatus]

:: DeveloperApi :: Return information about blocks stored in all of the slaves

def getLocalProperty(key: String): String

Get a local property set in this thread, or null if it is missing.

def getPersistentRDDs: Map[Int, RDD[_]]

Returns an immutable map of RDDs that have marked themselves as persistent via cache() call.

def getPoolForName(pool: String): Option[Schedulable]

:: DeveloperApi :: Return the pool associated with the given name, if one exists

def getRDDStorageInfo: Array[RDDInfo]

:: DeveloperApi :: Return information about what RDDs are cached, if they are in mem or on disk, how much space they take, etc.

def getSchedulingMode: SchedulingMode

Return current scheduling mode

def hadoopConfiguration: Configuration

A default Hadoop Configuration for the Hadoop code (e.g.

def hadoopFile[K, V, F <: InputFormat[K, V]](path: String)(implicit km: ClassTag[K], vm: ClassTag[V], fm: ClassTag[F]): RDD[(K, V)]

Smarter version of hadoopFile() that uses class tags to figure out the classes of keys, values and the InputFormat so that users don't need to pass them directly.

val file = sparkContext.hadoopFile[LongWritable, Text, TextInputFormat](path)

def hadoopFile[K, V, F <: InputFormat[K, V]](path: String, minPartitions: Int)(implicit km: ClassTag[K], vm: ClassTag[V], fm: ClassTag[F]): RDD[(K, V)]

Smarter version of hadoopFile() that uses class tags to figure out the classes of keys, values and the InputFormat so that users don't need to pass them directly.

val file = sparkContext.hadoopFile[LongWritable, Text, TextInputFormat](path, minPartitions)

def hadoopFile[K, V](path: String, inputFormatClass: Class[_ <: InputFormat[K, V]], keyClass: Class[K], valueClass: Class[V], minPartitions: Int = defaultMinPartitions): RDD[(K, V)]

Get an RDD for a Hadoop file with an arbitrary InputFormat

def hadoopRDD[K, V](conf: JobConf, inputFormatClass: Class[_ <: InputFormat[K, V]], keyClass: Class[K], valueClass: Class[V], minPartitions: Int = defaultMinPartitions): RDD[(K, V)]

Get an RDD for a Hadoop-readable dataset from a Hadoop JobConf given its InputFormat and other necessary info (e.g.

def hashCode(): Int

def initializeLogIfNecessary(isInterpreter: Boolean): Unit

final def isInstanceOf[T0]: Boolean

def isLocal: Boolean

def isStopped: Boolean

def isTraceEnabled(): Boolean

def jars: Seq[String]

def killExecutor(executorId: String): Boolean

:: DeveloperApi :: Request that the cluster manager kill the specified executor.

def killExecutors(executorIds: Seq[String]): Boolean

:: DeveloperApi :: Request that the cluster manager kill the specified executors.

def listFiles(): Seq[String]

Returns a list of file paths that are added to resources.

def listJars(): Seq[String]

Returns a list of jar files that are added to resources.

val localProperties: InheritableThreadLocal[Properties]

def log: Logger

def logDebug(msg: ⇒ String, throwable: Throwable): Unit

def logDebug(msg: ⇒ String): Unit

def logError(msg: ⇒ String, throwable: Throwable): Unit

def logError(msg: ⇒ String): Unit

def logInfo(msg: ⇒ String, throwable: Throwable): Unit

def logInfo(msg: ⇒ String): Unit

def logName: String

def logTrace(msg: ⇒ String, throwable: Throwable): Unit

def logTrace(msg: ⇒ String): Unit

def logWarning(msg: ⇒ String, throwable: Throwable): Unit

def logWarning(msg: ⇒ String): Unit

def longAccumulator(name: String): LongAccumulator

Create and register a long accumulator, which starts with 0 and accumulates inputs by add.

def longAccumulator: LongAccumulator

Create and register a long accumulator, which starts with 0 and accumulates inputs by add.

def makeRDD[T](seq: Seq[(T, Seq[String])])(implicit arg0: ClassTag[T]): RDD[T]

Distribute a local Scala collection to form an RDD, with one or more location preferences (hostnames of Spark nodes) for each object.

def makeRDD[T](seq: Seq[T], numSlices: Int = defaultParallelism)(implicit arg0: ClassTag[T]): RDD[T]

Distribute a local Scala collection to form an RDD.

def master: String

final def ne(arg0: AnyRef): Boolean

def newAPIHadoopFile[K, V, F <: InputFormat[K, V]](path: String, fClass: Class[F], kClass: Class[K], vClass: Class[V], conf: Configuration = hadoopConfiguration): RDD[(K, V)]

Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.

def newAPIHadoopFile[K, V, F <: InputFormat[K, V]](path: String)(implicit km: ClassTag[K], vm: ClassTag[V], fm: ClassTag[F]): RDD[(K, V)]

Get an RDD for a Hadoop file with an arbitrary new API InputFormat.

def newAPIHadoopRDD[K, V, F <: InputFormat[K, V]](conf: Configuration = hadoopConfiguration, fClass: Class[F], kClass: Class[K], vClass: Class[V]): RDD[(K, V)]

Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.

final def notify(): Unit

final def notifyAll(): Unit

def objectFile[T](path: String, minPartitions: Int = defaultMinPartitions)(implicit arg0: ClassTag[T]): RDD[T]

Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition.

def parallelize[T](seq: Seq[T], numSlices: Int = defaultParallelism)(implicit arg0: ClassTag[T]): RDD[T]

Distribute a local Scala collection to form an RDD.

def range(start: Long, end: Long, step: Long = 1, numSlices: Int = defaultParallelism): RDD[Long]

Creates a new RDD[Long] containing elements from start to end(exclusive), increased by step every element.

def register(acc: AccumulatorV2[_, _], name: String): Unit

Register the given accumulator with given name.

def register(acc: AccumulatorV2[_, _]): Unit

Register the given accumulator.

def requestExecutors(numAdditionalExecutors: Int): Boolean

:: DeveloperApi :: Request an additional number of executors from the cluster manager.

def requestTotalExecutors(numExecutors: Int, localityAwareTasks: Int, hostToLocalTaskCount: Map[String, Int]): Boolean

Update the cluster manager on our scheduling needs.

def runApproximateJob[T, U, R](rdd: RDD[T], func: (TaskContext, Iterator[T]) ⇒ U, evaluator: ApproximateEvaluator[U, R], timeout: Long): PartialResult[R]

:: DeveloperApi :: Run a job that can return approximate results.

def runJob[T, U](rdd: RDD[T], processPartition: (Iterator[T]) ⇒ U, resultHandler: (Int, U) ⇒ Unit)(implicit arg0: ClassTag[U]): Unit

Run a job on all partitions in an RDD and pass the results to a handler function.

def runJob[T, U](rdd: RDD[T], processPartition: (TaskContext, Iterator[T]) ⇒ U, resultHandler: (Int, U) ⇒ Unit)(implicit arg0: ClassTag[U]): Unit

Run a job on all partitions in an RDD and pass the results to a handler function.

def runJob[T, U](rdd: RDD[T], func: (Iterator[T]) ⇒ U)(implicit arg0: ClassTag[U]): Array[U]

Run a job on all partitions in an RDD and return the results in an array.

def runJob[T, U](rdd: RDD[T], func: (TaskContext, Iterator[T]) ⇒ U)(implicit arg0: ClassTag[U]): Array[U]

Run a job on all partitions in an RDD and return the results in an array.

def runJob[T, U](rdd: RDD[T], func: (Iterator[T]) ⇒ U, partitions: Seq[Int])(implicit arg0: ClassTag[U]): Array[U]

Run a job on a given set of partitions of an RDD, but take a function of type Iterator[T] => U instead of (TaskContext, Iterator[T]) => U.

def runJob[T, U](rdd: RDD[T], func: (TaskContext, Iterator[T]) ⇒ U, partitions: Seq[Int])(implicit arg0: ClassTag[U]): Array[U]

Run a function on a given set of partitions in an RDD and return the results as an array.

def runJob[T, U](rdd: RDD[T], func: (TaskContext, Iterator[T]) ⇒ U, partitions: Seq[Int], resultHandler: (Int, U) ⇒ Unit)(implicit arg0: ClassTag[U]): Unit

Run a function on a given set of partitions in an RDD and pass the results to the given handler function.

def sequenceFile[K, V](path: String, minPartitions: Int = defaultMinPartitions)(implicit km: ClassTag[K], vm: ClassTag[V], kcf: () ⇒ WritableConverter[K], vcf: () ⇒ WritableConverter[V]): RDD[(K, V)]

Version of sequenceFile() for types implicitly convertible to Writables through a WritableConverter.

sparkContext.sequenceFile[String, Int](path, ...)

def sequenceFile[K, V](path: String, keyClass: Class[K], valueClass: Class[V]): RDD[(K, V)]

Get an RDD for a Hadoop SequenceFile with given key and value types.

def sequenceFile[K, V](path: String, keyClass: Class[K], valueClass: Class[V], minPartitions: Int): RDD[(K, V)]

Get an RDD for a Hadoop SequenceFile with given key and value types.

def setCallSite(shortCallSite: String): Unit

Set the thread-local property for overriding the call sites of actions and RDDs.

def setCheckpointDir(directory: String): Unit

Set the directory under which RDDs are going to be checkpointed.

def setJobDescription(value: String): Unit

Set a human readable description of the current job.

def setJobGroup(groupId: String, description: String, interruptOnCancel: Boolean = false): Unit

Assigns a group ID to all the jobs started by this thread until the group ID is set to a different value or cleared.

// In the main thread:
sc.setJobGroup("some_job_to_cancel", "some job description")
sc.parallelize(1 to 10000, 2).map { i => Thread.sleep(10); i }.count()

// In a separate thread:
sc.cancelJobGroup("some_job_to_cancel")

def setLocalProperty(key: String, value: String): Unit

Set a local property that affects jobs submitted from this thread, such as the Spark fair scheduler pool.

def setLogLevel(logLevel: String): Unit

Control our logLevel.

val sparkUser: String

val startTime: Long

def statusTracker: SparkStatusTracker

def stop(): Unit

def submitJob[T, U, R](rdd: RDD[T], processPartition: (Iterator[T]) ⇒ U, partitions: Seq[Int], resultHandler: (Int, U) ⇒ Unit, resultFunc: ⇒ R): SimpleFutureAction[R]

Submit a job for execution and return a FutureJob holding the result.

final def synchronized[T0](arg0: ⇒ T0): T0

def textFile(path: String, minPartitions: Int = defaultMinPartitions): RDD[String]

Read a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings.

def toString(): String

def uiWebUrl: Option[String]

def union[T](first: RDD[T], rest: RDD[T]*)(implicit arg0: ClassTag[T]): RDD[T]

Build the union of a list of RDDs passed as variable-length arguments.

def union[T](rdds: Seq[RDD[T]])(implicit arg0: ClassTag[T]): RDD[T]

Build the union of a list of RDDs.

def version: String

The version of Spark on which this application is running.

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

def wholeTextFiles(path: String, minPartitions: Int = defaultMinPartitions): RDD[(String, String)]

Read a directory of text files from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI.

hdfs://a-hdfs-path/part-00000
hdfs://a-hdfs-path/part-00001
...
hdfs://a-hdfs-path/part-nnnnn

(a-hdfs-path/part-00000, its content)
(a-hdfs-path/part-00001, its content)
...
(a-hdfs-path/part-nnnnn, its content)

def accumulable[R, T](initialValue: R, name: String)(implicit param: AccumulableParam[R, T]): Accumulable[R, T]

Create an org.apache.spark.Accumulable shared variable, with a name for display in the Spark UI.

def accumulable[R, T](initialValue: R)(implicit param: AccumulableParam[R, T]): Accumulable[R, T]

Create an org.apache.spark.Accumulable shared variable, to which tasks can add values with +=.

def accumulableCollection[R, T](initialValue: R)(implicit arg0: (R) ⇒ Growable[T] with TraversableOnce[T] with Serializable, arg1: ClassTag[R]): Accumulable[R, T]

Create an accumulator from a "mutable collection" type.

def accumulator[T](initialValue: T, name: String)(implicit param: AccumulatorParam[T]): Accumulator[T]

Create an org.apache.spark.Accumulator variable of a given type, with a name for display in the Spark UI.

def accumulator[T](initialValue: T)(implicit param: AccumulatorParam[T]): Accumulator[T]

Create an org.apache.spark.Accumulator variable of a given type, which tasks can "add" values to using the += method.