org.apache.spark.sql.hive

Class HiveContext

Object

org.apache.spark.sql.SQLContext

org.apache.spark.sql.hive.HiveContext

All Implemented Interfaces:

java.io.Serializable, Logging, CacheManager, SQLConf, UDFRegistration

Direct Known Subclasses:

LocalHiveContext, TestHiveContext

public class HiveContext
extends SQLContext

An instance of the Spark SQL execution engine that integrates with data stored in Hive. Configuration for Hive is read from hive-site.xml on the classpath.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.spark.sql.SQLConf

SQLConf.Deprecated$

Constructor Summary

Constructors

Constructor and Description
HiveContext(SparkContext sc)

Method Summary

Methods

Modifier and Type	Method and Description
void	analyze(String tableName) Analyzes the given table in the current database to generate statistics, which will be used in query optimizations.
boolean	convertMetastoreParquet() When true, enables an experimental feature where metastore tables that use the parquet SerDe are automatically converted to use the Spark SQL parquet table scan, instead of the Hive SerDe.
<A extends scala.Product> void	createTable(String tableName, boolean allowExisting, scala.reflect.api.TypeTags.TypeTag<A> evidence$1) Creates a table using the schema of the given class.
String	dialect() The SQL dialect that is used when parsing queries.
org.apache.spark.sql.SQLContext.SparkPlanner	hivePlanner()
SchemaRDD	hiveql(String hqlQuery)
SchemaRDD	hql(String hqlQuery)
void	setConf(String key, String value) Set the given Spark SQL configuration property.
SchemaRDD	sql(String sqlText) Executes a SQL query using Spark, returning the result as a SchemaRDD.

Methods inherited from class org.apache.spark.sql.SQLContext

applySchema, applySchemaToPythonRDD, applySchemaToPythonRDD, baseRelationToSchemaRDD, createParquetFile, createSchemaRDD, dropTempTable, extraStrategies, jsonFile, jsonFile, jsonFile, jsonRDD, jsonRDD, jsonRDD, logicalPlanToSparkQuery, parquetFile, parseDataType, registerRDDAsTable, sparkContext, table

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Methods inherited from interface org.apache.spark.sql.SQLConf

autoBroadcastJoinThreshold, clear, codegenEnabled, columnBatchSize, columnNameOfCorruptRecord, defaultSizeInBytes, externalSortEnabled, getAllConfs, getConf, getConf, inMemoryPartitionPruning, isParquetBinaryAsString, numShufflePartitions, parquetCompressionCodec, parquetFilterPushDown, setConf, settings, useCompression

Methods inherited from interface org.apache.spark.sql.CacheManager

cachedData, cacheLock, cacheQuery, cacheTable, clearCache, invalidateCache, isCached, lookupCachedData, lookupCachedData, readLock, tryUncacheQuery, uncacheQuery, uncacheTable, useCachedData, writeLock

Methods inherited from interface org.apache.spark.sql.UDFRegistration

registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerFunction, registerPython

Constructor Detail

HiveContext

public HiveContext(SparkContext sc)

Method Detail

dialect

public String dialect()

Description copied from interface: SQLConf

The SQL dialect that is used when parsing queries. This defaults to 'sql' which uses a simple SQL parser provided by Spark SQL. This is currently the only option for users of SQLContext.

When using a HiveContext, this value defaults to 'hiveql', which uses the Hive 0.12.0 HiveQL parser. Users can change this to 'sql' if they want to run queries that aren't supported by HiveQL (e.g., SELECT 1).

Note that the choice of dialect does not affect things like what tables are available or how query execution is performed.

convertMetastoreParquet

public boolean convertMetastoreParquet()

When true, enables an experimental feature where metastore tables that use the parquet SerDe are automatically converted to use the Spark SQL parquet table scan, instead of the Hive SerDe.

sql

public SchemaRDD sql(String sqlText)

Description copied from class: SQLContext

Executes a SQL query using Spark, returning the result as a SchemaRDD. The dialect that is used for SQL parsing can be configured with 'spark.sql.dialect'.

Overrides:

sql in class SQLContext

hiveql

public SchemaRDD hiveql(String hqlQuery)

hql

public SchemaRDD hql(String hqlQuery)

createTable

public <A extends scala.Product> void createTable(String tableName,
                                         boolean allowExisting,
                                         scala.reflect.api.TypeTags.TypeTag<A> evidence$1)

Creates a table using the schema of the given class.

Parameters:

tableName - The name of the table to create.

allowExisting - When false, an exception will be thrown if the table already exists.

analyze

public void analyze(String tableName)

Analyzes the given table in the current database to generate statistics, which will be used in query optimizations.

Right now, it only supports Hive tables and it only updates the size of a Hive table in the Hive metastore.

setConf

public void setConf(String key,
           String value)

Description copied from interface: SQLConf

Set the given Spark SQL configuration property.

hivePlanner

public org.apache.spark.sql.SQLContext.SparkPlanner hivePlanner()