org.apache.spark.sql.connector.write

Interface RequiresDistributionAndOrdering

All Superinterfaces:

Write

@Experimental
public interface RequiresDistributionAndOrdering
extends Write

A write that requires a specific distribution and ordering of data.

Since:

3.2.0

Method Summary

Modifier and Type	Method and Description
default boolean	distributionStrictlyRequired() Returns if the distribution required by this write is strictly required or best effort only.
Distribution	requiredDistribution() Returns the distribution required by this write.
default int	requiredNumPartitions() Returns the number of partitions required by this write.
SortOrder[]	requiredOrdering() Returns the ordering required by this write.

Methods inherited from interface org.apache.spark.sql.connector.write.Write

description, supportedCustomMetrics, toBatch, toStreaming

Method Detail

requiredDistribution

Distribution requiredDistribution()

Returns the distribution required by this write.

Spark will distribute incoming records across partitions to satisfy the required distribution before passing the records to the data source table on write.

Batch and micro-batch writes can request a particular data distribution. If a distribution is requested in the micro-batch context, incoming records in each micro batch will satisfy the required distribution (but not across micro batches). The continuous execution mode continuously processes streaming data and does not support distribution requirements.

Implementations may return UnspecifiedDistribution if they don't require any specific distribution of data on write.

Returns:

the required distribution

distributionStrictlyRequired

default boolean distributionStrictlyRequired()

Returns if the distribution required by this write is strictly required or best effort only.

If true, Spark will strictly distribute incoming records across partitions to satisfy the required distribution before passing the records to the data source table on write. Otherwise, Spark may apply certain optimizations to speed up the query but break the distribution requirement.

Returns:

true if the distribution required by this write is strictly required; false otherwise.

requiredNumPartitions

default int requiredNumPartitions()

Returns the number of partitions required by this write.

Implementations may override this to require a specific number of input partitions.

Note that Spark doesn't support the number of partitions on UnspecifiedDistribution, the query will fail if the number of partitions are provided but the distribution is unspecified.

Returns:

the required number of partitions, any value less than 1 mean no requirement.

requiredOrdering

SortOrder[] requiredOrdering()

Returns the ordering required by this write.

Spark will order incoming records within partitions to satisfy the required ordering before passing those records to the data source table on write.

Batch and micro-batch writes can request a particular data ordering. If an ordering is requested in the micro-batch context, incoming records in each micro batch will satisfy the required ordering (but not across micro batches). The continuous execution mode continuously processes streaming data and does not support ordering requirements.

Implementations may return an empty array if they don't require any specific ordering of data on write.

Returns:

the required ordering