org.apache.spark.sql.sources.v2

Interface MicroBatchReadSupport

All Superinterfaces:

DataSourceV2

@InterfaceStability.Evolving
public interface MicroBatchReadSupport
extends DataSourceV2

A mix-in interface for DataSourceV2. Data sources can implement this interface to provide streaming micro-batch data reading ability.

Method Summary

Modifier and Type	Method and Description
MicroBatchReader	createMicroBatchReader(java.util.Optional<StructType> schema, String checkpointLocation, DataSourceOptions options) Creates a MicroBatchReader to read batches of data from this data source in a streaming query.

Method Detail

createMicroBatchReader

MicroBatchReader createMicroBatchReader(java.util.Optional<StructType> schema,
                                        String checkpointLocation,
                                        DataSourceOptions options)

Creates a MicroBatchReader to read batches of data from this data source in a streaming query. The execution engine will create a micro-batch reader at the start of a streaming query, alternate calls to setOffsetRange and planInputPartitions for each batch to process, and then call stop() when the execution is complete. Note that a single query may have multiple executions due to restart or failure recovery.

Parameters:

schema - the user provided schema, or empty() if none was provided

checkpointLocation - a path to Hadoop FS scratch space that can be used for failure recovery. Readers for the same logical source in the same query will be given the same checkpointLocation.

options - the options for the returned data source reader, which is an immutable case-insensitive string-to-string map.