org.apache.spark.api.plugin

Interface ExecutorPlugin

@DeveloperApi
public interface ExecutorPlugin

:: DeveloperApi :: Executor component of a SparkPlugin.

Since:

3.0.0

Method Summary

Modifier and Type	Method and Description
default void	init(PluginContext ctx, java.util.Map<String,String> extraConf) Initialize the executor plugin.
default void	onTaskFailed(TaskFailedReason failureReason) Perform an action after tasks completes with exceptions.
default void	onTaskStart() Perform any action before the task is run.
default void	onTaskSucceeded() Perform an action after tasks completes without exceptions.
default void	shutdown() Clean up and terminate this plugin.

Method Detail

init

default void init(PluginContext ctx,
                  java.util.Map<String,String> extraConf)

Initialize the executor plugin.

When a Spark plugin provides an executor plugin, this method will be called during the initialization of the executor process. It will block executor initialization until it returns.

Executor plugins that publish metrics should register all metrics with the context's registry (PluginContext.metricRegistry()) when this method is called. Metrics registered afterwards are not guaranteed to show up.

Parameters:

ctx - Context information for the executor where the plugin is running.

extraConf - Extra configuration provided by the driver component during its initialization.

shutdown

default void shutdown()

Clean up and terminate this plugin.

This method is called during the executor shutdown phase, and blocks executor shutdown.

onTaskStart

default void onTaskStart()

Perform any action before the task is run.

This method is invoked from the same thread the task will be executed. Task-specific information can be accessed via TaskContext.get().

Plugin authors should avoid expensive operations here, as this method will be called on every task, and doing something expensive can significantly slow down a job. It is not recommended for a user to call a remote service, for example.

Exceptions thrown from this method do not propagate - they're caught, logged, and suppressed. Therefore exceptions when executing this method won't make the job fail.

Since:

3.1.0

onTaskSucceeded

default void onTaskSucceeded()

Perform an action after tasks completes without exceptions.

As onTaskStart exceptions are suppressed, this method will still be invoked even if the corresponding onTaskStart() call for this task failed.

Same warnings of onTaskStart apply here.

Since:

3.1.0

onTaskFailed

default void onTaskFailed(TaskFailedReason failureReason)

Perform an action after tasks completes with exceptions.

Same warnings of onTaskStart apply here.

Parameters:

failureReason - the exception thrown from the failed task.

Since:

3.1.0