org.apache.spark.scheduler

Interface SchedulerBackend

public interface SchedulerBackend

A backend interface for scheduling systems that allows plugging in different ones under TaskSchedulerImpl. We assume a Mesos-like model where the application gets resource offers as machines become available and can launch tasks on them.

Method Summary

Modifier and Type	Method and Description
scala.Option<String>	applicationAttemptId() Get the attempt ID for this run, if the cluster manager supports multiple attempts.
String	applicationId() Get an application ID associated with the job.
int	defaultParallelism()
scala.Option<scala.collection.immutable.Map<String,String>>	getDriverAttributes() Get the attributes on driver.
scala.Option<scala.collection.immutable.Map<String,String>>	getDriverLogUrls() Get the URLs for the driver logs.
scala.collection.Seq<BlockManagerId>	getShufflePushMergerLocations(int numPartitions, int resourceProfileId) Get the list of host locations for push based shuffle
boolean	isReady()
void	killTask(long taskId, String executorId, boolean interruptThread, String reason) Requests that an executor kills a running task.
int	maxNumConcurrentTasks(ResourceProfile rp) Get the max number of tasks that can be concurrent launched based on the ResourceProfile could be used, even if some of them are being used at the moment.
void	reviveOffers() Update the current offers and schedule tasks
void	start()
void	stop()

Method Detail

applicationAttemptId

scala.Option<String> applicationAttemptId()

Get the attempt ID for this run, if the cluster manager supports multiple attempts. Applications run in client mode will not have attempt IDs.

Returns:

The application attempt id, if available.

applicationId

String applicationId()

Get an application ID associated with the job.

Returns:

An application ID

defaultParallelism

int defaultParallelism()

getDriverAttributes

scala.Option<scala.collection.immutable.Map<String,String>> getDriverAttributes()

Get the attributes on driver. These attributes are used to replace log URLs when custom log url pattern is specified.

Returns:

Map containing attributes on driver.

getDriverLogUrls

scala.Option<scala.collection.immutable.Map<String,String>> getDriverLogUrls()

Get the URLs for the driver logs. These URLs are used to display the links in the UI Executors tab for the driver.

Returns:

Map containing the log names and their respective URLs

getShufflePushMergerLocations

scala.collection.Seq<BlockManagerId> getShufflePushMergerLocations(int numPartitions,
                                                                   int resourceProfileId)

Get the list of host locations for push based shuffle

Currently push based shuffle is disabled for both stage retry and stage reuse cases (for eg: in the case where few partitions are lost due to failure). Hence this method should be invoked only once for a ShuffleDependency.

Parameters:

numPartitions - (undocumented)

resourceProfileId - (undocumented)

Returns:

List of external shuffle services locations

isReady

boolean isReady()

killTask

void killTask(long taskId,
              String executorId,
              boolean interruptThread,
              String reason)

Requests that an executor kills a running task.

Parameters:

taskId - Id of the task.

executorId - Id of the executor the task is running on.

interruptThread - Whether the executor should interrupt the task thread.

reason - The reason for the task kill.

maxNumConcurrentTasks

int maxNumConcurrentTasks(ResourceProfile rp)

Get the max number of tasks that can be concurrent launched based on the ResourceProfile could be used, even if some of them are being used at the moment. Note that please don't cache the value returned by this method, because the number can change due to add/remove executors.

Parameters:

rp - ResourceProfile which to use to calculate max concurrent tasks.

Returns:

The max number of tasks that can be concurrent launched currently.

reviveOffers

void reviveOffers()

Update the current offers and schedule tasks

start

void start()

stop

void stop()