org.apache.spark.sql

Interface SchemaRDDLike

All Known Implementing Classes:

JavaSchemaRDD, SchemaRDD

public interface SchemaRDDLike

Contains functions that are shared between all SchemaRDD types (i.e., Scala, Java)

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan	baseLogicalPlan()
SchemaRDD	baseSchemaRDD()
void	insertInto(String tableName) :: Experimental :: Appends the rows from this RDD to the specified table.
void	insertInto(String tableName, boolean overwrite) :: Experimental :: Adds the rows from this RDD to the specified table, optionally overwriting the existing data.
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan	logicalPlan()
void	printSchema() Prints out the schema.
org.apache.spark.sql.SQLContext.QueryExecution	queryExecution() :: DeveloperApi :: A lazily computed query execution workflow.
void	registerAsTable(String tableName)
void	registerTempTable(String tableName) Registers this RDD as a temporary table using the given name.
void	saveAsParquetFile(String path) Saves the contents of this SchemaRDD as a parquet file, preserving the schema.
void	saveAsTable(String tableName) :: Experimental :: Creates a table from the the contents of this SchemaRDD.
String	schemaString() Returns the schema as a string in the tree format.
SQLContext	sqlContext()
String	toString()

Method Detail

sqlContext

SQLContext sqlContext()

baseLogicalPlan

org.apache.spark.sql.catalyst.plans.logical.LogicalPlan baseLogicalPlan()

baseSchemaRDD

SchemaRDD baseSchemaRDD()

queryExecution

org.apache.spark.sql.SQLContext.QueryExecution queryExecution()

:: DeveloperApi :: A lazily computed query execution workflow. All other RDD operations are passed through to the RDD that is produced by this workflow. This workflow is produced lazily because invoking the whole query optimization pipeline can be expensive.

The query execution is considered a Developer API as phases may be added or removed in future releases. This execution is only exposed to provide an interface for inspecting the various phases for debugging purposes. Applications should not depend on particular phases existing or producing any specific output, even for exactly the same query.

Additionally, the RDD exposed by this execution is not designed for consumption by end users. In particular, it does not contain any schema information, and it reuses Row objects internally. This object reuse improves performance, but can make programming against the RDD more difficult. Instead end users should perform RDD operations on a SchemaRDD directly.

logicalPlan

org.apache.spark.sql.catalyst.plans.logical.LogicalPlan logicalPlan()

toString

String toString()

Overrides:

toString in class Object

saveAsParquetFile

void saveAsParquetFile(String path)

Saves the contents of this SchemaRDD as a parquet file, preserving the schema. Files that are written out using this method can be read back in as a SchemaRDD using the parquetFile function.

registerTempTable

void registerTempTable(String tableName)

Registers this RDD as a temporary table using the given name. The lifetime of this temporary table is tied to the SQLContext that was used to create this SchemaRDD.

registerAsTable

void registerAsTable(String tableName)

insertInto

void insertInto(String tableName,
              boolean overwrite)

:: Experimental :: Adds the rows from this RDD to the specified table, optionally overwriting the existing data.

insertInto

void insertInto(String tableName)

:: Experimental :: Appends the rows from this RDD to the specified table.

saveAsTable

void saveAsTable(String tableName)

:: Experimental :: Creates a table from the the contents of this SchemaRDD. This will fail if the table already exists.

Note that this currently only works with SchemaRDDs that are created from a HiveContext as there is no notion of a persisted catalog in a standard SQL context. Instead you can write an RDD out to a parquet file, and then register that file as a table. This "table" can then be the target of an insertInto.

schemaString

String schemaString()

Returns the schema as a string in the tree format.

printSchema

void printSchema()

Prints out the schema.