org.apache.spark.streaming.scheduler

Class ReceivedBlockTracker

Object

org.apache.spark.streaming.scheduler.ReceivedBlockTracker

All Implemented Interfaces:

Logging

public class ReceivedBlockTracker
extends Object
implements Logging

Class that keep track of all the received blocks, and allocate them to batches when required. All actions taken by this class can be saved to a write ahead log (if a checkpoint directory has been provided), so that the state of the tracker (received blocks and block-to-batch allocations) can be recovered after driver failure.

Note that when any instance of this class is created with a checkpoint directory, it will try reading events from logs in the directory.

Constructor Summary

Constructors

Constructor and Description
ReceivedBlockTracker(SparkConf conf, org.apache.hadoop.conf.Configuration hadoopConf, scala.collection.Seq<Object> streamIds, Clock clock, scala.Option<String> checkpointDirOption)

Method Summary

Methods

Modifier and Type	Method and Description
boolean	addBlock(ReceivedBlockInfo receivedBlockInfo) Add received block.
void	allocateBlocksToBatch(Time batchTime) Allocate all unallocated blocks to the given batch.
static String	checkpointDirToLogDir(String checkpointDir)
void	cleanupOldBatches(Time cleanupThreshTime) Clean up block information of old batches.
scala.collection.immutable.Map<Object,scala.collection.Seq<ReceivedBlockInfo>>	getBlocksOfBatch(Time batchTime) Get the blocks allocated to the given batch.
scala.collection.Seq<ReceivedBlockInfo>	getBlocksOfBatchAndStream(Time batchTime, int streamId) Get the blocks allocated to the given batch and stream.
scala.collection.Seq<ReceivedBlockInfo>	getUnallocatedBlocks(int streamId) Get blocks that have been added but not yet allocated to any batch.
boolean	hasUnallocatedReceivedBlocks() Check if any blocks are left to be allocated to batches.
boolean	isLogManagerEnabled() Check if the log manager is enabled.
void	stop() Stop the block tracker.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

ReceivedBlockTracker

public ReceivedBlockTracker(SparkConf conf,
                    org.apache.hadoop.conf.Configuration hadoopConf,
                    scala.collection.Seq<Object> streamIds,
                    Clock clock,
                    scala.Option<String> checkpointDirOption)

Method Detail

checkpointDirToLogDir

public static String checkpointDirToLogDir(String checkpointDir)

addBlock

public boolean addBlock(ReceivedBlockInfo receivedBlockInfo)

Add received block. This event will get written to the write ahead log (if enabled).

allocateBlocksToBatch

public void allocateBlocksToBatch(Time batchTime)

Allocate all unallocated blocks to the given batch. This event will get written to the write ahead log (if enabled).

getBlocksOfBatch

public scala.collection.immutable.Map<Object,scala.collection.Seq<ReceivedBlockInfo>> getBlocksOfBatch(Time batchTime)

Get the blocks allocated to the given batch.

getBlocksOfBatchAndStream

public scala.collection.Seq<ReceivedBlockInfo> getBlocksOfBatchAndStream(Time batchTime,
                                                                int streamId)

Get the blocks allocated to the given batch and stream.

hasUnallocatedReceivedBlocks

public boolean hasUnallocatedReceivedBlocks()

Check if any blocks are left to be allocated to batches.

getUnallocatedBlocks

public scala.collection.Seq<ReceivedBlockInfo> getUnallocatedBlocks(int streamId)

Get blocks that have been added but not yet allocated to any batch. This method is primarily used for testing.

cleanupOldBatches

public void cleanupOldBatches(Time cleanupThreshTime)

Clean up block information of old batches.

stop

public void stop()

Stop the block tracker.

isLogManagerEnabled

public boolean isLogManagerEnabled()

Check if the log manager is enabled. This is only used for testing purposes.