SparkSession
Related Docs:
object SparkSession
| package sql
Convert a BaseRelation created for external data sources into a DataFrame.
Convert a BaseRelation created for external data sources into a DataFrame.
2.0.0
Interface through which the user may create, drop, alter or query underlying databases, tables, functions etc.
Interface through which the user may create, drop, alter or query underlying databases, tables, functions etc.
2.0.0
Runtime configuration interface for Spark.
Runtime configuration interface for Spark.
This is the interface through which the user can get and set all Spark and Hadoop configurations that are relevant to Spark SQL. When getting the value of a config, this defaults to the value set in the underlying SparkContext, if any.
2.0.0
Applies a schema to a List of Java Beans.
Applies a schema to a List of Java Beans.
WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.
1.6.0
Applies a schema to an RDD of Java Beans.
Applies a schema to an RDD of Java Beans.
WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.
2.0.0
Applies a schema to an RDD of Java Beans.
Applies a schema to an RDD of Java Beans.
WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.
2.0.0
:: DeveloperApi :: Creates a DataFrame from a java.util.List containing Rows using the given schema.
:: DeveloperApi :: Creates a DataFrame from a JavaRDD containing Rows using the given schema.
:: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema.
:: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception. Example:
import org.apache.spark.sql._ import org.apache.spark.sql.types._ val sparkSession = new org.apache.spark.sql.SparkSession(sc) val schema = StructType( StructField("name", StringType, false) :: StructField("age", IntegerType, true) :: Nil) val people = sc.textFile("examples/src/main/resources/people.txt").map( _.split(",")).map(p => Row(p(0), p(1).trim.toInt)) val dataFrame = sparkSession.createDataFrame(people, schema) dataFrame.printSchema // root // |-- name: string (nullable = false) // |-- age: integer (nullable = true) dataFrame.createOrReplaceTempView("people") sparkSession.sql("select name from people").collect.foreach(println)
2.0.0
:: Experimental :: Creates a DataFrame from a local Seq of Product.
:: Experimental :: Creates a DataFrame from a local Seq of Product.
2.0.0
:: Experimental :: Creates a DataFrame from an RDD of Product (e.g.
:: Experimental :: Creates a DataFrame from an RDD of Product (e.g. case classes, tuples).
2.0.0
:: Experimental :: Creates a Dataset from a java.util.List of a given type.
:: Experimental ::
Creates a Dataset from a java.util.List of a given type. This method requires an
encoder (to convert a JVM object of type T to and from the internal Spark SQL representation)
that is generally created automatically through implicits from a SparkSession, or can be
created explicitly by calling static methods on Encoders.
List<String> data = Arrays.asList("hello", "world"); Dataset<String> ds = spark.createDataset(data, Encoders.STRING());
2.0.0
:: Experimental :: Creates a Dataset from an RDD of a given type.
:: Experimental ::
Creates a Dataset from an RDD of a given type. This method requires an
encoder (to convert a JVM object of type T to and from the internal Spark SQL representation)
that is generally created automatically through implicits from a SparkSession, or can be
created explicitly by calling static methods on Encoders.
2.0.0
:: Experimental :: Creates a Dataset from a local Seq of data of a given type.
:: Experimental ::
Creates a Dataset from a local Seq of data of a given type. This method requires an
encoder (to convert a JVM object of type T to and from the internal Spark SQL representation)
that is generally created automatically through implicits from a SparkSession, or can be
created explicitly by calling static methods on Encoders.
import spark.implicits._ case class Person(name: String, age: Long) val data = Seq(Person("Michael", 29), Person("Andy", 30), Person("Justin", 19)) val ds = spark.createDataset(data) ds.show() // +-------+---+ // | name|age| // +-------+---+ // |Michael| 29| // | Andy| 30| // | Justin| 19| // +-------+---+
2.0.0
Returns a DataFrame with no rows or columns.
Returns a DataFrame with no rows or columns.
2.0.0
:: Experimental :: Creates a new Dataset of type T containing zero elements.
:: Experimental :: Creates a new Dataset of type T containing zero elements.
2.0.0
:: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.
:: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.
2.0.0
:: Experimental :: (Scala-specific) Implicit methods available in Scala for converting common Scala objects into DataFrames.
:: Experimental :: (Scala-specific) Implicit methods available in Scala for converting common Scala objects into DataFrames.
val sparkSession = SparkSession.builder.getOrCreate() import sparkSession.implicits._
2.0.0
:: Experimental :: An interface to register custom org.apache.spark.sql.util.QueryExecutionListeners that listen for execution metrics.
:: Experimental :: An interface to register custom org.apache.spark.sql.util.QueryExecutionListeners that listen for execution metrics.
2.0.0
Start a new session with isolated SQL configurations, temporary tables, registered functions are isolated, but sharing the underlying SparkContext and cached data.
Start a new session with isolated SQL configurations, temporary tables, registered functions are isolated, but sharing the underlying SparkContext and cached data.
Note: Other than the SparkContext, all shared state is initialized lazily. This method will force the initialization of the shared state to ensure that parent and child sessions are set up with the same shared state. If the underlying catalog implementation is Hive, this will initialize the metastore, which may take some time.
2.0.0
Parses the data type in our internal string representation.
Parses the data type in our internal string representation. The data type string should
have the same format as the one generated by toString in scala.
It is only used by PySpark.
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in a range from start to end (exclusive) with a step value, with partition number
specified.
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in a range from start to end (exclusive) with a step value, with partition number
specified.
2.0.0
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in a range from start to end (exclusive) with a step value.
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in a range from start to end (exclusive) with a step value.
2.0.0
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in a range from start to end (exclusive) with step value 1.
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in a range from start to end (exclusive) with step value 1.
2.0.0
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in a range from 0 to end (exclusive) with step value 1.
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in a range from 0 to end (exclusive) with step value 1.
2.0.0
Returns a DataFrameReader that can be used to read non-streaming data in as a DataFrame.
Returns a DataFrameReader that can be used to read non-streaming data in as a DataFrame.
sparkSession.read.parquet("/path/to/file.parquet") sparkSession.read.schema(schema).json("/path/to/file.json")
2.0.0
:: Experimental :: Returns a DataStreamReader that can be used to read streaming data in as a DataFrame.
:: Experimental :: Returns a DataStreamReader that can be used to read streaming data in as a DataFrame.
sparkSession.readStream.parquet("/path/to/directory/of/parquet/files") sparkSession.readStream.schema(schema).json("/path/to/directory/of/json/files")
2.0.0
Executes a SQL query using Spark, returning the result as a DataFrame.
Executes a SQL query using Spark, returning the result as a DataFrame. The dialect that is used for SQL parsing can be configured with 'spark.sql.dialect'.
2.0.0
A wrapped version of this session in the form of a SQLContext, for backward compatibility.
A wrapped version of this session in the form of a SQLContext, for backward compatibility.
2.0.0
Stop the underlying SparkContext.
Stop the underlying SparkContext.
2.0.0
:: Experimental ::
Returns a StreamingQueryManager that allows managing all the
StreamingQueries active on this.
:: Experimental ::
Returns a StreamingQueryManager that allows managing all the
StreamingQueries active on this.
2.0.0
Returns the specified table as a DataFrame.
Returns the specified table as a DataFrame.
2.0.0
A collection of methods for registering user-defined functions (UDF).
A collection of methods for registering user-defined functions (UDF). Note that the user-defined functions must be deterministic. Due to optimization, duplicate invocations may be eliminated or the function may even be invoked more times than it is present in the query.
The following example registers a Scala closure as UDF:
sparkSession.udf.register("myUDF", (arg1: Int, arg2: String) => arg2 + arg1)
The following example registers a UDF in Java:
sparkSession.udf().register("myUDF", new UDF2<Integer, String, String>() { @Override public String call(Integer arg1, String arg2) { return arg2 + arg1; } }, DataTypes.StringType);
Or, to use Java 8 lambda syntax:
sparkSession.udf().register("myUDF", (Integer arg1, String arg2) -> arg2 + arg1, DataTypes.StringType);
2.0.0
The version of Spark on which this application is running.
The version of Spark on which this application is running.
2.0.0
The entry point to programming Spark with the Dataset and DataFrame API.
In environments that this has been created upfront (e.g. REPL, notebooks), use the builder to get an existing session:
The builder can also be used to create a new session: