org.apache.spark.sql.sources.v2

Class DataSourceOptions

Object

org.apache.spark.sql.sources.v2.DataSourceOptions

@InterfaceStability.Evolving
public class DataSourceOptions
extends Object

An immutable string-to-string map in which keys are case-insensitive. This is used to represent data source options. Each data source implementation can define its own options and teach its users how to set them. Spark doesn't have any restrictions about what options a data source should or should not have. Instead Spark defines some standard options that data sources can optionally adopt. It's possible that some options are very common and many data sources use them. However different data sources may define the common options(key and meaning) differently, which is quite confusing to end users. The standard options defined by Spark:

Option key	Option value
path	A path string of the data files/directories, like path1, /absolute/file2, path3/*. The path can either be relative or absolute, points to either file or directory, and can contain wildcards. This option is commonly used by file-based data sources.
paths	A JSON array style paths string of the data files/directories, like ["path1", "/absolute/file2"]. The format of each path is same as the path option, plus it should follow JSON string literal format, e.g. quotes should be escaped, pa\"th means pa"th.
table	A table name string representing the table name directly without any interpretation. For example, db.tbl means a table called db.tbl, not a table called tbl inside database db. `t*b.l` means a table called `t*b.l`, not t*b.l.
database	A database name string representing the database name directly without any interpretation, which is very similar to the table name option.

Field Summary

Fields

Modifier and Type	Field and Description
static String	DATABASE_KEY The option key for database name.
static String	PATH_KEY The option key for singular path.
static String	PATHS_KEY The option key for multiple paths.
static String	TABLE_KEY The option key for table name.

Constructor Summary

Constructors

Constructor and Description
DataSourceOptions(java.util.Map<String,String> originalMap)

Method Summary

Modifier and Type	Method and Description
java.util.Map<String,String>	asMap()
java.util.Optional<String>	databaseName() Returns the value of the database name option.
static DataSourceOptions	empty()
java.util.Optional<String>	get(String key) Returns the option value to which the specified key is mapped, case-insensitively.
boolean	getBoolean(String key, boolean defaultValue) Returns the boolean value to which the specified key is mapped, or defaultValue if there is no mapping for the key.
double	getDouble(String key, double defaultValue) Returns the double value to which the specified key is mapped, or defaultValue if there is no mapping for the key.
int	getInt(String key, int defaultValue) Returns the integer value to which the specified key is mapped, or defaultValue if there is no mapping for the key.
long	getLong(String key, long defaultValue) Returns the long value to which the specified key is mapped, or defaultValue if there is no mapping for the key.
String[]	paths() Returns all the paths specified by both the singular path option and the multiple paths option.
java.util.Optional<String>	tableName() Returns the value of the table name option.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PATH_KEY

public static final String PATH_KEY

The option key for singular path.

See Also:

Constant Field Values

PATHS_KEY

public static final String PATHS_KEY

The option key for multiple paths.

See Also:

Constant Field Values

TABLE_KEY

public static final String TABLE_KEY

The option key for table name.

See Also:

Constant Field Values

DATABASE_KEY

public static final String DATABASE_KEY

The option key for database name.

See Also:

Constant Field Values

Constructor Detail

DataSourceOptions

public DataSourceOptions(java.util.Map<String,String> originalMap)

Method Detail

empty

public static DataSourceOptions empty()

asMap

public java.util.Map<String,String> asMap()

get

public java.util.Optional<String> get(String key)

Returns the option value to which the specified key is mapped, case-insensitively.

getBoolean

public boolean getBoolean(String key,
                          boolean defaultValue)

Returns the boolean value to which the specified key is mapped, or defaultValue if there is no mapping for the key. The key match is case-insensitive

getInt

public int getInt(String key,
                  int defaultValue)

Returns the integer value to which the specified key is mapped, or defaultValue if there is no mapping for the key. The key match is case-insensitive

getLong

public long getLong(String key,
                    long defaultValue)

Returns the long value to which the specified key is mapped, or defaultValue if there is no mapping for the key. The key match is case-insensitive

getDouble

public double getDouble(String key,
                        double defaultValue)

Returns the double value to which the specified key is mapped, or defaultValue if there is no mapping for the key. The key match is case-insensitive

paths

public String[] paths()

Returns all the paths specified by both the singular path option and the multiple paths option.

tableName

public java.util.Optional<String> tableName()

Returns the value of the table name option.

databaseName

public java.util.Optional<String> databaseName()

Returns the value of the database name option.