org.apache.spark.resource

Class ExecutorResourceRequests

Object

org.apache.spark.resource.ExecutorResourceRequests

All Implemented Interfaces:

java.io.Serializable

public class ExecutorResourceRequests
extends Object
implements scala.Serializable

A set of Executor resource requests. This is used in conjunction with the ResourceProfile to programmatically specify the resources needed for an RDD that will be applied at the stage level.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
ExecutorResourceRequests()

Method Summary

Modifier and Type	Method and Description
ExecutorResourceRequests	cores(int amount) Specify number of cores per Executor.
ExecutorResourceRequests	memory(String amount) Specify heap memory.
ExecutorResourceRequests	memoryOverhead(String amount) Specify overhead memory.
ExecutorResourceRequests	offHeapMemory(String amount) Specify off heap memory.
ExecutorResourceRequests	pysparkMemory(String amount) Specify pyspark memory.
scala.collection.immutable.Map<String,ExecutorResourceRequest>	requests() Returns all the resource requests for the task.
java.util.Map<String,ExecutorResourceRequest>	requestsJMap() (Java-specific) Returns all the resource requests for the executor.
ExecutorResourceRequests	resource(String resourceName, long amount, String discoveryScript, String vendor) Amount of a particular custom resource(GPU, FPGA, etc) to use.
String	toString()

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

ExecutorResourceRequests

public ExecutorResourceRequests()

Method Detail

cores

public ExecutorResourceRequests cores(int amount)

Specify number of cores per Executor. This is a convenient API to add ExecutorResourceRequest for "cores" resource.

Parameters:

amount - Number of cores to allocate per Executor.

Returns:

(undocumented)

memory

public ExecutorResourceRequests memory(String amount)

Specify heap memory. The value specified will be converted to MiB. This is a convenient API to add ExecutorResourceRequest for "memory" resource.

Parameters:

amount - Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.

Returns:

(undocumented)

memoryOverhead

public ExecutorResourceRequests memoryOverhead(String amount)

Specify overhead memory. The value specified will be converted to MiB. This is a convenient API to add ExecutorResourceRequest for "memoryOverhead" resource.

Parameters:

amount - Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.

Returns:

(undocumented)

offHeapMemory

public ExecutorResourceRequests offHeapMemory(String amount)

Specify off heap memory. The value specified will be converted to MiB. This value only take effect when MEMORY_OFFHEAP_ENABLED is true. This is a convenient API to add ExecutorResourceRequest for "offHeap" resource.

Parameters:

amount - Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.

Returns:

(undocumented)

pysparkMemory

public ExecutorResourceRequests pysparkMemory(String amount)

Specify pyspark memory. The value specified will be converted to MiB. This is a convenient API to add ExecutorResourceRequest for "pyspark.memory" resource.

Parameters:

amount - Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.

Returns:

(undocumented)

requests

public scala.collection.immutable.Map<String,ExecutorResourceRequest> requests()

Returns all the resource requests for the task.

Returns:

(undocumented)

requestsJMap

public java.util.Map<String,ExecutorResourceRequest> requestsJMap()

(Java-specific) Returns all the resource requests for the executor.

Returns:

(undocumented)

resource

public ExecutorResourceRequests resource(String resourceName,
                                         long amount,
                                         String discoveryScript,
                                         String vendor)

Amount of a particular custom resource(GPU, FPGA, etc) to use. The resource names supported correspond to the regular Spark configs with the prefix removed. For instance, resources like GPUs are gpu (spark configs spark.executor.resource.gpu.*). If you pass in a resource that the cluster manager doesn't support the result is undefined, it may error or may just be ignored. This is a convenient API to add ExecutorResourceRequest for custom resources.

Parameters:

resourceName - Name of the resource.

amount - amount of that resource per executor to use.

discoveryScript - Optional script used to discover the resources. This is required on some cluster managers that don't tell Spark the addresses of the resources allocated. The script runs on Executors startup to of the resources available.

vendor - Optional vendor, required for some cluster managers

Returns:

(undocumented)

toString

public String toString()

Overrides:

toString in class Object