org.apache.spark.storage

Class DiskBlockManager

Object

org.apache.spark.storage.DiskBlockManager

All Implemented Interfaces:

Logging

public class DiskBlockManager
extends Object
implements Logging

Creates and maintains the logical mapping between logical blocks and physical on-disk locations. By default, one block is mapped to one file with a name given by its BlockId. However, it is also possible to have a block map to only a segment of a file, by calling mapBlockToFileSegment().

Block files are hashed among the directories listed in spark.local.dir (or in SPARK_LOCAL_DIRS, if it's set).

Constructor Summary

Constructors

Constructor and Description
DiskBlockManager(BlockManager blockManager, SparkConf conf)

Method Summary

Methods

Modifier and Type	Method and Description
boolean	containsBlock(BlockId blockId) Check if disk block manager has a block.
scala.Tuple2<TempLocalBlockId,java.io.File>	createTempLocalBlock() Produces a unique block id and File suitable for storing local intermediate results.
scala.Tuple2<TempShuffleBlockId,java.io.File>	createTempShuffleBlock() Produces a unique block id and File suitable for storing shuffled intermediate results.
scala.collection.Seq<BlockId>	getAllBlocks() List all the blocks currently stored on disk by the disk manager.
scala.collection.Seq<java.io.File>	getAllFiles() List all the files currently stored on disk by the disk manager.
java.io.File	getFile(BlockId blockId)
java.io.File	getFile(String filename) Looks up a file by hashing it into one of our local subdirectories.
java.io.File[]	localDirs()
void	stop() Cleanup local dirs and stop shuffle sender.
int	subDirsPerLocalDir()

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

DiskBlockManager

public DiskBlockManager(BlockManager blockManager,
                SparkConf conf)

Method Detail

subDirsPerLocalDir

public int subDirsPerLocalDir()

localDirs

public java.io.File[] localDirs()

getFile

public java.io.File getFile(String filename)

Looks up a file by hashing it into one of our local subdirectories.

getFile

public java.io.File getFile(BlockId blockId)

containsBlock

public boolean containsBlock(BlockId blockId)

Check if disk block manager has a block.

getAllFiles

public scala.collection.Seq<java.io.File> getAllFiles()

List all the files currently stored on disk by the disk manager.

getAllBlocks

public scala.collection.Seq<BlockId> getAllBlocks()

List all the blocks currently stored on disk by the disk manager.

createTempLocalBlock

public scala.Tuple2<TempLocalBlockId,java.io.File> createTempLocalBlock()

Produces a unique block id and File suitable for storing local intermediate results.

createTempShuffleBlock

public scala.Tuple2<TempShuffleBlockId,java.io.File> createTempShuffleBlock()

Produces a unique block id and File suitable for storing shuffled intermediate results.

stop

public void stop()

Cleanup local dirs and stop shuffle sender.