org.apache.spark.sql.connector.write

Interface WriteBuilder

All Known Subinterfaces:

SupportsDynamicOverwrite, SupportsOverwrite, SupportsTruncate, V1WriteBuilder

@Evolving
public interface WriteBuilder

An interface for building the BatchWrite. Implementations can mix in some interfaces to support different ways to write data to data sources. Unless modified by a mixin interface, the BatchWrite configured by this builder is to append data without affecting existing data.

Method Summary

Modifier and Type	Method and Description
default BatchWrite	buildForBatch() Returns a BatchWrite to write data to batch source.
default StreamingWrite	buildForStreaming() Returns a StreamingWrite to write data to streaming source.
default WriteBuilder	withInputDataSchema(StructType schema) Passes the schema of the input data from Spark to data source.
default WriteBuilder	withQueryId(String queryId) Passes the `queryId` from Spark to data source.

Method Detail

withQueryId

default WriteBuilder withQueryId(String queryId)

Passes the `queryId` from Spark to data source. `queryId` is a unique string of the query. It's possible that there are many queries running at the same time, or a query is restarted and resumed. BatchWrite can use this id to identify the query.

Returns:

a new builder with the `queryId`. By default it returns `this`, which means the given `queryId` is ignored. Please override this method to take the `queryId`.

withInputDataSchema

default WriteBuilder withInputDataSchema(StructType schema)

Passes the schema of the input data from Spark to data source.

Returns:

a new builder with the `schema`. By default it returns `this`, which means the given `schema` is ignored. Please override this method to take the `schema`.

buildForBatch

default BatchWrite buildForBatch()

Returns a BatchWrite to write data to batch source. By default this method throws exception, data sources must overwrite this method to provide an implementation, if the Table that creates this write returns TableCapability.BATCH_WRITE support in its Table.capabilities().

buildForStreaming

default StreamingWrite buildForStreaming()

Returns a StreamingWrite to write data to streaming source. By default this method throws exception, data sources must overwrite this method to provide an implementation, if the Table that creates this write returns TableCapability.STREAMING_WRITE support in its Table.capabilities().