org.apache.spark

Interface ExecutorAllocationClient

public interface ExecutorAllocationClient

A client that communicates with the cluster manager to request or kill executors. This is currently supported only in YARN mode.

Method Summary

Modifier and Type	Method and Description
scala.collection.Seq<String>	getExecutorIds() Get the list of currently active executors
boolean	killExecutor(String executorId) Request that the cluster manager kill the specified executor.
scala.collection.Seq<String>	killExecutors(scala.collection.Seq<String> executorIds, boolean adjustTargetNumExecutors, boolean countFailures, boolean force) Request that the cluster manager kill the specified executors.
boolean	killExecutorsOnHost(String host) Request that the cluster manager kill every executor on the specified host.
boolean	requestExecutors(int numAdditionalExecutors) Request an additional number of executors from the cluster manager.
boolean	requestTotalExecutors(int numExecutors, int localityAwareTasks, scala.collection.immutable.Map<String,Object> hostToLocalTaskCount) Update the cluster manager on our scheduling needs.

Method Detail

getExecutorIds

scala.collection.Seq<String> getExecutorIds()

Get the list of currently active executors

killExecutor

boolean killExecutor(String executorId)

Request that the cluster manager kill the specified executor.

Parameters:

executorId - (undocumented)

Returns:

whether the request is acknowledged by the cluster manager.

killExecutors

scala.collection.Seq<String> killExecutors(scala.collection.Seq<String> executorIds,
                                           boolean adjustTargetNumExecutors,
                                           boolean countFailures,
                                           boolean force)

Request that the cluster manager kill the specified executors.

Parameters:

executorIds - identifiers of executors to kill

adjustTargetNumExecutors - whether the target number of executors will be adjusted down after these executors have been killed

countFailures - if there are tasks running on the executors when they are killed, whether to count those failures toward task failure limits

force - whether to force kill busy executors, default false

Returns:

the ids of the executors acknowledged by the cluster manager to be removed.

killExecutorsOnHost

boolean killExecutorsOnHost(String host)

Request that the cluster manager kill every executor on the specified host.

Parameters:

host - (undocumented)

Returns:

whether the request is acknowledged by the cluster manager.

requestExecutors

boolean requestExecutors(int numAdditionalExecutors)

Request an additional number of executors from the cluster manager.

Parameters:

numAdditionalExecutors - (undocumented)

Returns:

whether the request is acknowledged by the cluster manager.

requestTotalExecutors

boolean requestTotalExecutors(int numExecutors,
                              int localityAwareTasks,
                              scala.collection.immutable.Map<String,Object> hostToLocalTaskCount)

Update the cluster manager on our scheduling needs. Three bits of information are included to help it make decisions.

Parameters:

numExecutors - The total number of executors we'd like to have. The cluster manager shouldn't kill any running executor to reach this number, but, if all existing executors were to die, this is the number of executors we'd want to be allocated.

localityAwareTasks - The number of tasks in all active stages that have a locality preferences. This includes running, pending, and completed tasks.

hostToLocalTaskCount - A map of hosts to the number of tasks from all active stages that would like to like to run on that host. This includes running, pending, and completed tasks.

Returns:

whether the request is acknowledged by the cluster manager.