SparkLauncher (Spark 2.0.0 JavaDoc)

Object
- org.apache.spark.launcher.SparkLauncher

```
public class SparkLauncher
extends Object
```
Launcher for Spark applications.
Use this class to start Spark applications programmatically. The class uses a builder pattern to allow clients to configure the Spark application and launch it as a child process.

Field Summary

Fields
Modifier and Type	Field and Description
`static String`	`CHILD_CONNECTION_TIMEOUT` Maximum time (in ms) to wait for a child process to connect back to the launcher server when using @link{#start()}.
`static String`	`CHILD_PROCESS_LOGGER_NAME` Logger name to use when launching a child process.
`static String`	`DEPLOY_MODE` The Spark deploy mode.
`static String`	`DRIVER_EXTRA_CLASSPATH` Configuration key for the driver class path.
`static String`	`DRIVER_EXTRA_JAVA_OPTIONS` Configuration key for the driver VM options.
`static String`	`DRIVER_EXTRA_LIBRARY_PATH` Configuration key for the driver native library path.
`static String`	`DRIVER_MEMORY` Configuration key for the driver memory.
`static String`	`EXECUTOR_CORES` Configuration key for the number of executor CPU cores.
`static String`	`EXECUTOR_EXTRA_CLASSPATH` Configuration key for the executor class path.
`static String`	`EXECUTOR_EXTRA_JAVA_OPTIONS` Configuration key for the executor VM options.
`static String`	`EXECUTOR_EXTRA_LIBRARY_PATH` Configuration key for the executor native library path.
`static String`	`EXECUTOR_MEMORY` Configuration key for the executor memory.
`static String`	`NO_RESOURCE` A special value for the resource that tells Spark to not try to process the app resource as a file.
`static String`	`SPARK_MASTER` The Spark master.

Constructor Summary

Constructors
Constructor and Description

SparkLauncher()

SparkLauncher(java.util.Map<String,String> env)
Creates a launcher that will set the given environment variables in the child.

Constructors
Constructor and Description
`SparkLauncher()`
`SparkLauncher(java.util.Map<String,String> env)` Creates a launcher that will set the given environment variables in the child.

Method Summary

Methods
Modifier and Type	Method and Description
`SparkLauncher`	`addAppArgs(String... args)` Adds command line arguments for the application.
`SparkLauncher`	`addFile(String file)` Adds a file to be submitted with the application.
`SparkLauncher`	`addJar(String jar)` Adds a jar file to be submitted with the application.
`SparkLauncher`	`addPyFile(String file)` Adds a python file / zip / egg to be submitted with the application.
`SparkLauncher`	`addSparkArg(String arg)` Adds a no-value argument to the Spark invocation.
`SparkLauncher`	`addSparkArg(String name, String value)` Adds an argument with a value to the Spark invocation.
`Process`	`launch()` Launches a sub-process that will start the configured Spark application.
`SparkLauncher`	`setAppName(String appName)` Set the application name.
`SparkLauncher`	`setAppResource(String resource)` Set the main application resource.
`SparkLauncher`	`setConf(String key, String value)` Set a single configuration value for the application.
`static void`	`setConfig(String name, String value)` Set a configuration value for the launcher library.
`SparkLauncher`	`setDeployMode(String mode)` Set the deploy mode for the application.
`SparkLauncher`	`setJavaHome(String javaHome)` Set a custom JAVA_HOME for launching the Spark application.
`SparkLauncher`	`setMainClass(String mainClass)` Sets the application class name for Java/Scala applications.
`SparkLauncher`	`setMaster(String master)` Set the Spark master for the application.
`SparkLauncher`	`setPropertiesFile(String path)` Set a custom properties file with Spark configuration for the application.
`SparkLauncher`	`setSparkHome(String sparkHome)` Set a custom Spark installation location for the application.
`SparkLauncher`	`setVerbose(boolean verbose)` Enables verbose reporting for SparkSubmit.
`SparkAppHandle`	`startApplication(SparkAppHandle.Listener... listeners)` Starts a Spark application.

Methods inherited from class Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - SPARK_MASTER
```
public static final String SPARK_MASTER
```
    The Spark master.
    
    See Also:
    Constant Field Values
  - DEPLOY_MODE
```
public static final String DEPLOY_MODE
```
    The Spark deploy mode.
    
    See Also:
    Constant Field Values
  - DRIVER_MEMORY
```
public static final String DRIVER_MEMORY
```
    Configuration key for the driver memory.
    
    See Also:
    Constant Field Values
  - DRIVER_EXTRA_CLASSPATH
```
public static final String DRIVER_EXTRA_CLASSPATH
```
    Configuration key for the driver class path.
    
    See Also:
    Constant Field Values
  - DRIVER_EXTRA_JAVA_OPTIONS
```
public static final String DRIVER_EXTRA_JAVA_OPTIONS
```
    Configuration key for the driver VM options.
    
    See Also:
    Constant Field Values
  - DRIVER_EXTRA_LIBRARY_PATH
```
public static final String DRIVER_EXTRA_LIBRARY_PATH
```
    Configuration key for the driver native library path.
    
    See Also:
    Constant Field Values
  - EXECUTOR_MEMORY
```
public static final String EXECUTOR_MEMORY
```
    Configuration key for the executor memory.
    
    See Also:
    Constant Field Values
  - EXECUTOR_EXTRA_CLASSPATH
```
public static final String EXECUTOR_EXTRA_CLASSPATH
```
    Configuration key for the executor class path.
    
    See Also:
    Constant Field Values
  - EXECUTOR_EXTRA_JAVA_OPTIONS
```
public static final String EXECUTOR_EXTRA_JAVA_OPTIONS
```
    Configuration key for the executor VM options.
    
    See Also:
    Constant Field Values
  - EXECUTOR_EXTRA_LIBRARY_PATH
```
public static final String EXECUTOR_EXTRA_LIBRARY_PATH
```
    Configuration key for the executor native library path.
    
    See Also:
    Constant Field Values
  - EXECUTOR_CORES
```
public static final String EXECUTOR_CORES
```
    Configuration key for the number of executor CPU cores.
    
    See Also:
    Constant Field Values
  - CHILD_PROCESS_LOGGER_NAME
```
public static final String CHILD_PROCESS_LOGGER_NAME
```
    Logger name to use when launching a child process.
    
    See Also:
    Constant Field Values
  - NO_RESOURCE
```
public static final String NO_RESOURCE
```
    A special value for the resource that tells Spark to not try to process the app resource as a file. This is useful when the class being executed is added to the application using other means - for example, by adding jars using the package download feature.
    
    See Also:
    Constant Field Values
  - CHILD_CONNECTION_TIMEOUT
```
public static final String CHILD_CONNECTION_TIMEOUT
```
    Maximum time (in ms) to wait for a child process to connect back to the launcher server when using @link{#start()}.
    
    See Also:
    Constant Field Values
- Constructor Detail
  - SparkLauncher
```
public SparkLauncher()
```
  - SparkLauncher
```
public SparkLauncher(java.util.Map<String,String> env)
```
    Creates a launcher that will set the given environment variables in the child.
    
    Parameters:
    env - Environment variables to set.
- Method Detail
  - setConfig
```
public static void setConfig(String name,
             String value)
```
    Set a configuration value for the launcher library. These config values do not affect the launched application, but rather the behavior of the launcher library itself when managing applications.
    
    Parameters:
    name - Config name.
    value - Config value.
    Since:
    
    1.6.0
  - setJavaHome
```
public SparkLauncher setJavaHome(String javaHome)
```
    Set a custom JAVA_HOME for launching the Spark application.
    
    Parameters:
    javaHome - Path to the JAVA_HOME to use.
    
    Returns:
    This launcher.
  - setSparkHome
```
public SparkLauncher setSparkHome(String sparkHome)
```
    Set a custom Spark installation location for the application.
    
    Parameters:
    sparkHome - Path to the Spark installation to use.
    
    Returns:
    This launcher.
  - setPropertiesFile
```
public SparkLauncher setPropertiesFile(String path)
```
    Set a custom properties file with Spark configuration for the application.
    
    Parameters:
    path - Path to custom properties file to use.
    
    Returns:
    This launcher.
  - setConf
```
public SparkLauncher setConf(String key,
                    String value)
```
    Set a single configuration value for the application.
    
    Parameters:
    key - Configuration key.
    value - The value to use.
    
    Returns:
    This launcher.
  - setAppName
```
public SparkLauncher setAppName(String appName)
```
    Set the application name.
    
    Parameters:
    appName - Application name.
    
    Returns:
    This launcher.
  - setMaster
```
public SparkLauncher setMaster(String master)
```
    Set the Spark master for the application.
    
    Parameters:
    master - Spark master.
    
    Returns:
    This launcher.
  - setDeployMode
```
public SparkLauncher setDeployMode(String mode)
```
    Set the deploy mode for the application.
    
    Parameters:
    mode - Deploy mode.
    
    Returns:
    This launcher.
  - setAppResource
```
public SparkLauncher setAppResource(String resource)
```
    Set the main application resource. This should be the location of a jar file for Scala/Java applications, or a python script for PySpark applications.
    
    Parameters:
    resource - Path to the main application resource.
    
    Returns:
    This launcher.
  - setMainClass
```
public SparkLauncher setMainClass(String mainClass)
```
    Sets the application class name for Java/Scala applications.
    
    Parameters:
    mainClass - Application's main class.
    
    Returns:
    This launcher.
  - addSparkArg
```
public SparkLauncher addSparkArg(String arg)
```
    Adds a no-value argument to the Spark invocation. If the argument is known, this method validates whether the argument is indeed a no-value argument, and throws an exception otherwise.
    Use this method with caution. It is possible to create an invalid Spark command by passing unknown arguments to this method, since those are allowed for forward compatibility.
    
    Parameters:
    arg - Argument to add.
    
    Returns:
    This launcher.
    Since:
    
    1.5.0
  - addSparkArg
```
public SparkLauncher addSparkArg(String name,
                        String value)
```
    Adds an argument with a value to the Spark invocation. If the argument name corresponds to a known argument, the code validates that the argument actually expects a value, and throws an exception otherwise.
    It is safe to add arguments modified by other methods in this class (such as setMaster(String) - the last invocation will be the one to take effect.
    Use this method with caution. It is possible to create an invalid Spark command by passing unknown arguments to this method, since those are allowed for forward compatibility.
    
    Parameters:
    name - Name of argument to add.
    value - Value of the argument.
    
    Returns:
    This launcher.
    Since:
    
    1.5.0
  - addAppArgs
```
public SparkLauncher addAppArgs(String... args)
```
    Adds command line arguments for the application.
    
    Parameters:
    args - Arguments to pass to the application's main class.
    
    Returns:
    This launcher.
  - addJar
```
public SparkLauncher addJar(String jar)
```
    Adds a jar file to be submitted with the application.
    
    Parameters:
    jar - Path to the jar file.
    
    Returns:
    This launcher.
  - addFile
```
public SparkLauncher addFile(String file)
```
    Adds a file to be submitted with the application.
    
    Parameters:
    file - Path to the file.
    
    Returns:
    This launcher.
  - addPyFile
```
public SparkLauncher addPyFile(String file)
```
    Adds a python file / zip / egg to be submitted with the application.
    
    Parameters:
    file - Path to the file.
    
    Returns:
    This launcher.
  - setVerbose
```
public SparkLauncher setVerbose(boolean verbose)
```
    Enables verbose reporting for SparkSubmit.
    
    Parameters:
    verbose - Whether to enable verbose output.
    
    Returns:
    This launcher.
  - launch
```
public Process launch()
               throws java.io.IOException
```
    Launches a sub-process that will start the configured Spark application.
    The startApplication(SparkAppHandle.Listener...) method is preferred when launching Spark, since it provides better control of the child application.
    
    Returns:
    A process handle for the Spark app.
    
    Throws:
    
    java.io.IOException
  - startApplication
```
public SparkAppHandle startApplication(SparkAppHandle.Listener... listeners)
                                throws java.io.IOException
```
    Starts a Spark application.
    This method returns a handle that provides information about the running application and can be used to do basic interaction with it.
    The returned handle assumes that the application will instantiate a single SparkContext during its lifetime. Once that context reports a final state (one that indicates the SparkContext has stopped), the handle will not perform new state transitions, so anything that happens after that cannot be monitored. If the underlying application is launched as a child process, SparkAppHandle.kill() can still be used to kill the child process.
    Currently, all applications are launched as child processes. The child's stdout and stderr are merged and written to a logger (see java.util.logging). The logger's name can be defined by setting CHILD_PROCESS_LOGGER_NAME in the app's configuration. If that option is not set, the code will try to derive a name from the application's name or main class / script file. If those cannot be determined, an internal, unique name will be used. In all cases, the logger name will start with "org.apache.spark.launcher.app", to fit more easily into the configuration of commonly-used logging systems.
    
    Parameters:
    listeners - Listeners to add to the handle before the app is launched.
    
    Returns:
    A handle for the launched application.
    
    Throws:
    
    java.io.IOException
    Since:
    
    1.6.0

Class SparkLauncher

Field Summary

Constructor Summary

Method Summary

Methods inherited from class Object

Field Detail

SPARK_MASTER

DEPLOY_MODE

DRIVER_MEMORY

DRIVER_EXTRA_CLASSPATH

DRIVER_EXTRA_JAVA_OPTIONS

DRIVER_EXTRA_LIBRARY_PATH

EXECUTOR_MEMORY

EXECUTOR_EXTRA_CLASSPATH

EXECUTOR_EXTRA_JAVA_OPTIONS

EXECUTOR_EXTRA_LIBRARY_PATH

EXECUTOR_CORES

CHILD_PROCESS_LOGGER_NAME

NO_RESOURCE

CHILD_CONNECTION_TIMEOUT

Constructor Detail

SparkLauncher

SparkLauncher

Method Detail

setConfig

setJavaHome

setSparkHome

setPropertiesFile

setConf

setAppName

setMaster

setDeployMode

setAppResource

setMainClass

addSparkArg

addSparkArg

addAppArgs

addJar

addFile

addPyFile

setVerbose

launch

startApplication