org.apache.spark.sql

Interface CacheManager

All Known Implementing Classes:

HiveContext, LocalHiveContext, SQLContext, TestHive, TestHiveContext, TestSQLContext

public interface CacheManager

Provides support in a SQLContext for caching query results and automatically using these cached results when subsequent queries are executed. Data is cached using byte buffers stored in an InMemoryRelation. This relation is automatically substituted query plans that return the sameResult as the originally cached query.

Method Summary

Methods

Modifier and Type	Method and Description
scala.collection.mutable.ArrayBuffer<CachedData>	cachedData()
java.util.concurrent.locks.ReentrantReadWriteLock	cacheLock()
void	cacheQuery(SchemaRDD query, scala.Option<String> tableName, StorageLevel storageLevel) Caches the data produced by the logical representation of the given schema rdd.
void	cacheTable(String tableName) Caches the specified table in-memory.
void	clearCache()
void	invalidateCache(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan) Invalidates the cache of any data that contains plan.
boolean	isCached(String tableName) Returns true if the table is currently cached in-memory.
scala.Option<CachedData>	lookupCachedData(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan) Optionally returns cached data for the given LogicalPlan.
scala.Option<CachedData>	lookupCachedData(SchemaRDD query) Optionally returns cached data for the given SchemaRDD
<A> A	readLock(scala.Function0<A> f) Acquires a read lock on the cache for the duration of `f`.
boolean	tryUncacheQuery(SchemaRDD query, boolean blocking) Tries to remove the data for the given SchemaRDD from the cache if it's cached
void	uncacheQuery(SchemaRDD query, boolean blocking) Removes the data for the given SchemaRDD from the cache
void	uncacheTable(String tableName) Removes the specified table from the in-memory cache.
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan	useCachedData(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan) Replaces segments of the given logical plan with cached versions where possible.
<A> A	writeLock(scala.Function0<A> f) Acquires a write lock on the cache for the duration of `f`.

Method Detail

cachedData

scala.collection.mutable.ArrayBuffer<CachedData> cachedData()

cacheLock

java.util.concurrent.locks.ReentrantReadWriteLock cacheLock()

isCached

boolean isCached(String tableName)

Returns true if the table is currently cached in-memory.

cacheTable

void cacheTable(String tableName)

Caches the specified table in-memory.

uncacheTable

void uncacheTable(String tableName)

Removes the specified table from the in-memory cache.

readLock

<A> A readLock(scala.Function0<A> f)

Acquires a read lock on the cache for the duration of `f`.

writeLock

<A> A writeLock(scala.Function0<A> f)

Acquires a write lock on the cache for the duration of `f`.

clearCache

void clearCache()

cacheQuery

void cacheQuery(SchemaRDD query,
              scala.Option<String> tableName,
              StorageLevel storageLevel)

Caches the data produced by the logical representation of the given schema rdd. Unlike RDD.cache(), the default storage level is set to be MEMORY_AND_DISK because recomputing the in-memory columnar representation of the underlying table is expensive.

uncacheQuery

void uncacheQuery(SchemaRDD query,
                boolean blocking)

Removes the data for the given SchemaRDD from the cache

tryUncacheQuery

boolean tryUncacheQuery(SchemaRDD query,
                      boolean blocking)

Tries to remove the data for the given SchemaRDD from the cache if it's cached

lookupCachedData

scala.Option<CachedData> lookupCachedData(SchemaRDD query)

Optionally returns cached data for the given SchemaRDD

lookupCachedData

scala.Option<CachedData> lookupCachedData(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan)

Optionally returns cached data for the given LogicalPlan.

useCachedData

org.apache.spark.sql.catalyst.plans.logical.LogicalPlan useCachedData(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan)

Replaces segments of the given logical plan with cached versions where possible.

invalidateCache

void invalidateCache(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan)

Invalidates the cache of any data that contains plan. Note that it is possible that this function will over invalidate.