org.apache.spark.sql.catalog

Class Catalog

Object

org.apache.spark.sql.catalog.Catalog

public abstract class Catalog
extends Object

Catalog interface for Spark. To access this, use SparkSession.catalog.

Since:

2.0.0

Constructor Summary

Constructors

Constructor and Description
Catalog()

Method Summary

Methods

Modifier and Type	Method and Description
abstract void	cacheTable(String tableName) Caches the specified table in-memory.
abstract void	clearCache() Removes all cached tables from the in-memory cache.
abstract Dataset<Row>	createExternalTable(String tableName, String path) :: Experimental :: Creates an external table from the given path and returns the corresponding DataFrame.
abstract Dataset<Row>	createExternalTable(String tableName, String source, java.util.Map<String,String> options) :: Experimental :: Creates an external table from the given path based on a data source and a set of options.
abstract Dataset<Row>	createExternalTable(String tableName, String source, scala.collection.immutable.Map<String,String> options) :: Experimental :: (Scala-specific) Creates an external table from the given path based on a data source and a set of options.
abstract Dataset<Row>	createExternalTable(String tableName, String path, String source) :: Experimental :: Creates an external table from the given path based on a data source and returns the corresponding DataFrame.
abstract Dataset<Row>	createExternalTable(String tableName, String source, StructType schema, java.util.Map<String,String> options) :: Experimental :: Create an external table from the given path based on a data source, a schema and a set of options.
abstract Dataset<Row>	createExternalTable(String tableName, String source, StructType schema, scala.collection.immutable.Map<String,String> options) :: Experimental :: (Scala-specific) Create an external table from the given path based on a data source, a schema and a set of options.
abstract String	currentDatabase() Returns the current default database in this session.
abstract boolean	databaseExists(String dbName) Check if the database with the specified name exists.
abstract boolean	dropGlobalTempView(String viewName) Drops the global temporary view with the given view name in the catalog.
abstract boolean	dropTempView(String viewName) Drops the local temporary view with the given view name in the catalog.
abstract boolean	functionExists(String functionName) Check if the function with the specified name exists.
abstract boolean	functionExists(String dbName, String functionName) Check if the function with the specified name exists in the specified database.
abstract Database	getDatabase(String dbName) Get the database with the specified name.
abstract Function	getFunction(String functionName) Get the function with the specified name.
abstract Function	getFunction(String dbName, String functionName) Get the function with the specified name.
abstract Table	getTable(String tableName) Get the table or view with the specified name.
abstract Table	getTable(String dbName, String tableName) Get the table or view with the specified name in the specified database.
abstract boolean	isCached(String tableName) Returns true if the table is currently cached in-memory.
abstract Dataset<Column>	listColumns(String tableName) Returns a list of columns for the given table in the current database or the given temporary table.
abstract Dataset<Column>	listColumns(String dbName, String tableName) Returns a list of columns for the given table in the specified database.
abstract Dataset<Database>	listDatabases() Returns a list of databases available across all sessions.
abstract Dataset<Function>	listFunctions() Returns a list of functions registered in the current database.
abstract Dataset<Function>	listFunctions(String dbName) Returns a list of functions registered in the specified database.
abstract Dataset<Table>	listTables() Returns a list of tables in the current database.
abstract Dataset<Table>	listTables(String dbName) Returns a list of tables in the specified database.
abstract void	recoverPartitions(String tableName) Recover all the partitions in the directory of a table and update the catalog.
abstract void	refreshByPath(String path) Invalidate and refresh all the cached data (and the associated metadata) for any dataframe that contains the given data source path.
abstract void	refreshTable(String tableName) Invalidate and refresh all the cached metadata of the given table.
abstract void	setCurrentDatabase(String dbName) Sets the current default database in this session.
abstract boolean	tableExists(String tableName) Check if the table or view with the specified name exists.
abstract boolean	tableExists(String dbName, String tableName) Check if the table or view with the specified name exists in the specified database.
abstract void	uncacheTable(String tableName) Removes the specified table from the in-memory cache.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Catalog

public Catalog()

Method Detail

currentDatabase

public abstract String currentDatabase()

Returns the current default database in this session.

Returns:

(undocumented)

Since:

2.0.0

setCurrentDatabase

public abstract void setCurrentDatabase(String dbName)

Sets the current default database in this session.

Parameters:

dbName - (undocumented)

Since:

2.0.0

listDatabases

public abstract Dataset<Database> listDatabases()

Returns a list of databases available across all sessions.

Returns:

(undocumented)

Since:

2.0.0

listTables

public abstract Dataset<Table> listTables()

Returns a list of tables in the current database. This includes all temporary tables.

Returns:

(undocumented)

Since:

2.0.0

listTables

public abstract Dataset<Table> listTables(String dbName)
                                   throws AnalysisException

Returns a list of tables in the specified database. This includes all temporary tables.

Parameters:

dbName - (undocumented)

Returns:

(undocumented)

Throws:

AnalysisException

Since:

2.0.0

listFunctions

public abstract Dataset<Function> listFunctions()

Returns a list of functions registered in the current database. This includes all temporary functions

Returns:

(undocumented)

Since:

2.0.0

listFunctions

public abstract Dataset<Function> listFunctions(String dbName)
                                         throws AnalysisException

Returns a list of functions registered in the specified database. This includes all temporary functions

Parameters:

dbName - (undocumented)

Returns:

(undocumented)

Throws:

AnalysisException

Since:

2.0.0

listColumns

public abstract Dataset<Column> listColumns(String tableName)
                                     throws AnalysisException

Returns a list of columns for the given table in the current database or the given temporary table.

Parameters:

tableName - (undocumented)

Returns:

(undocumented)

Throws:

AnalysisException

Since:

2.0.0

listColumns

public abstract Dataset<Column> listColumns(String dbName,
                          String tableName)
                                     throws AnalysisException

Returns a list of columns for the given table in the specified database.

Parameters:

dbName - (undocumented)

tableName - (undocumented)

Returns:

(undocumented)

Throws:

AnalysisException

Since:

2.0.0

getDatabase

public abstract Database getDatabase(String dbName)
                              throws AnalysisException

Get the database with the specified name. This throws an AnalysisException when the database cannot be found.

Parameters:

dbName - (undocumented)

Returns:

(undocumented)

Throws:

AnalysisException

Since:

2.1.0

getTable

public abstract Table getTable(String tableName)
                        throws AnalysisException

Get the table or view with the specified name. This table can be a temporary view or a table/view in the current database. This throws an AnalysisException when no Table can be found.

Parameters:

tableName - (undocumented)

Returns:

(undocumented)

Throws:

AnalysisException

Since:

2.1.0

getTable

public abstract Table getTable(String dbName,
             String tableName)
                        throws AnalysisException

Get the table or view with the specified name in the specified database. This throws an AnalysisException when no Table can be found.

Parameters:

dbName - (undocumented)

tableName - (undocumented)

Returns:

(undocumented)

Throws:

AnalysisException

Since:

2.1.0

getFunction

public abstract Function getFunction(String functionName)
                              throws AnalysisException

Get the function with the specified name. This function can be a temporary function or a function in the current database. This throws an AnalysisException when the function cannot be found.

Parameters:

functionName - (undocumented)

Returns:

(undocumented)

Throws:

AnalysisException

Since:

2.1.0

getFunction

public abstract Function getFunction(String dbName,
                   String functionName)
                              throws AnalysisException

Get the function with the specified name. This throws an AnalysisException when the function cannot be found.

Parameters:

dbName - (undocumented)

functionName - (undocumented)

Returns:

(undocumented)

Throws:

AnalysisException

Since:

2.1.0

databaseExists

public abstract boolean databaseExists(String dbName)

Check if the database with the specified name exists.

Parameters:

dbName - (undocumented)

Returns:

(undocumented)

Since:

2.1.0

tableExists

public abstract boolean tableExists(String tableName)

Check if the table or view with the specified name exists. This can either be a temporary view or a table/view in the current database.

Parameters:

tableName - (undocumented)

Returns:

(undocumented)

Since:

2.1.0

tableExists

public abstract boolean tableExists(String dbName,
                  String tableName)

Check if the table or view with the specified name exists in the specified database.

Parameters:

dbName - (undocumented)

tableName - (undocumented)

Returns:

(undocumented)

Since:

2.1.0

functionExists

public abstract boolean functionExists(String functionName)

Check if the function with the specified name exists. This can either be a temporary function or a function in the current database.

Parameters:

functionName - (undocumented)

Returns:

(undocumented)

Since:

2.1.0

functionExists

public abstract boolean functionExists(String dbName,
                     String functionName)

Check if the function with the specified name exists in the specified database.

Parameters:

dbName - (undocumented)

functionName - (undocumented)

Returns:

(undocumented)

Since:

2.1.0

createExternalTable

public abstract Dataset<Row> createExternalTable(String tableName,
                               String path)

:: Experimental :: Creates an external table from the given path and returns the corresponding DataFrame. It will use the default data source configured by spark.sql.sources.default.

Parameters:

tableName - (undocumented)

path - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createExternalTable

public abstract Dataset<Row> createExternalTable(String tableName,
                               String path,
                               String source)

:: Experimental :: Creates an external table from the given path based on a data source and returns the corresponding DataFrame.

Parameters:

tableName - (undocumented)

path - (undocumented)

source - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createExternalTable

public abstract Dataset<Row> createExternalTable(String tableName,
                               String source,
                               java.util.Map<String,String> options)

:: Experimental :: Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

Parameters:

tableName - (undocumented)

source - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createExternalTable

public abstract Dataset<Row> createExternalTable(String tableName,
                               String source,
                               scala.collection.immutable.Map<String,String> options)

:: Experimental :: (Scala-specific) Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

Parameters:

tableName - (undocumented)

source - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createExternalTable

public abstract Dataset<Row> createExternalTable(String tableName,
                               String source,
                               StructType schema,
                               java.util.Map<String,String> options)

:: Experimental :: Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

Parameters:

tableName - (undocumented)

source - (undocumented)

schema - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createExternalTable

public abstract Dataset<Row> createExternalTable(String tableName,
                               String source,
                               StructType schema,
                               scala.collection.immutable.Map<String,String> options)

:: Experimental :: (Scala-specific) Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

Parameters:

tableName - (undocumented)

source - (undocumented)

schema - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

dropTempView

public abstract boolean dropTempView(String viewName)

Drops the local temporary view with the given view name in the catalog. If the view has been cached before, then it will also be uncached.

Local temporary view is session-scoped. Its lifetime is the lifetime of the session that created it, i.e. it will be automatically dropped when the session terminates. It's not tied to any databases, i.e. we can't use db1.view1 to reference a local temporary view.

Note that, the return type of this method was Unit in Spark 2.0, but changed to Boolean in Spark 2.1.

Parameters:

viewName - the name of the view to be dropped.

Returns:

true if the view is dropped successfully, false otherwise.

Since:

2.0.0

dropGlobalTempView

public abstract boolean dropGlobalTempView(String viewName)

Drops the global temporary view with the given view name in the catalog. If the view has been cached before, then it will also be uncached.

Global temporary view is cross-session. Its lifetime is the lifetime of the Spark application, i.e. it will be automatically dropped when the application terminates. It's tied to a system preserved database _global_temp, and we must use the qualified name to refer a global temp view, e.g. SELECT * FROM _global_temp.view1.

Parameters:

viewName - the name of the view to be dropped.

Returns:

true if the view is dropped successfully, false otherwise.

Since:

2.1.0

recoverPartitions

public abstract void recoverPartitions(String tableName)

Recover all the partitions in the directory of a table and update the catalog.

Parameters:

tableName - (undocumented)

Since:

2.1.1

isCached

public abstract boolean isCached(String tableName)

Returns true if the table is currently cached in-memory.

Parameters:

tableName - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

cacheTable

public abstract void cacheTable(String tableName)

Caches the specified table in-memory.

Parameters:

tableName - (undocumented)

Since:

2.0.0

uncacheTable

public abstract void uncacheTable(String tableName)

Removes the specified table from the in-memory cache.

Parameters:

tableName - (undocumented)

Since:

2.0.0

clearCache

public abstract void clearCache()

Removes all cached tables from the in-memory cache.

Since:

2.0.0

refreshTable

public abstract void refreshTable(String tableName)

Invalidate and refresh all the cached metadata of the given table. For performance reasons, Spark SQL or the external data source library it uses might cache certain metadata about a table, such as the location of blocks. When those change outside of Spark SQL, users should call this function to invalidate the cache.

If this table is cached as an InMemoryRelation, drop the original cached version and make the new version cached lazily.

Parameters:

tableName - (undocumented)

Since:

2.0.0

refreshByPath

public abstract void refreshByPath(String path)

Invalidate and refresh all the cached data (and the associated metadata) for any dataframe that contains the given data source path. Path matching is by prefix, i.e. "/" would invalidate everything that is cached.

Parameters:

path - (undocumented)

Since:

2.0.0