org.apache.spark.sql.expressions

Class Aggregator<IN,BUF,OUT>

Object

org.apache.spark.sql.expressions.Aggregator<IN,BUF,OUT>

All Implemented Interfaces:

java.io.Serializable

public abstract class Aggregator<IN,BUF,OUT>
extends Object
implements scala.Serializable

:: Experimental :: A base class for user-defined aggregations, which can be used in Dataset operations to take all of the elements of a group and reduce them to a single value.

For example, the following aggregator extracts an int from a specific class and adds them up:

   case class Data(i: Int)

   val customSummer =  new Aggregator[Data, Int, Int] {
     def zero: Int = 0
     def reduce(b: Int, a: Data): Int = b + a.i
     def merge(b1: Int, b2: Int): Int = b1 + b2
     def finish(r: Int): Int = r
   }.toColumn()

   val ds: Dataset[Data] = ...
   val aggregated = ds.select(customSummer)
 

Based loosely on Aggregator from Algebird: https://github.com/twitter/algebird

Since:

1.6.0

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
Aggregator()

Method Summary

Methods

Modifier and Type	Method and Description
abstract Encoder<BUF>	bufferEncoder() Specifies the Encoder for the intermediate value type.
abstract OUT	finish(BUF reduction) Transform the output of the reduction.
abstract BUF	merge(BUF b1, BUF b2) Merge two intermediate values.
abstract Encoder<OUT>	outputEncoder() Specifies the Encoder for the final ouput value type.
abstract BUF	reduce(BUF b, IN a) Combine two values to produce a new value.
TypedColumn<IN,OUT>	toColumn() Returns this Aggregator as a TypedColumn that can be used in Dataset.
abstract BUF	zero() A zero value for this aggregation.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Aggregator

public Aggregator()

Method Detail

zero

public abstract BUF zero()

A zero value for this aggregation. Should satisfy the property that any b + zero = b.

Returns:

(undocumented)

Since:

1.6.0

reduce

public abstract BUF reduce(BUF b,
         IN a)

Combine two values to produce a new value. For performance, the function may modify b and return it instead of constructing new object for b.

Parameters:

b - (undocumented)

a - (undocumented)

Returns:

(undocumented)

Since:

1.6.0

merge

public abstract BUF merge(BUF b1,
        BUF b2)

Merge two intermediate values.

Parameters:

b1 - (undocumented)

b2 - (undocumented)

Returns:

(undocumented)

Since:

1.6.0

finish

public abstract OUT finish(BUF reduction)

Transform the output of the reduction.

Parameters:

reduction - (undocumented)

Returns:

(undocumented)

Since:

1.6.0

bufferEncoder

public abstract Encoder<BUF> bufferEncoder()

Specifies the Encoder for the intermediate value type.

Returns:

(undocumented)

Since:

2.0.0

outputEncoder

public abstract Encoder<OUT> outputEncoder()

Specifies the Encoder for the final ouput value type.

Returns:

(undocumented)

Since:

2.0.0

toColumn

public TypedColumn<IN,OUT> toColumn()

Returns this Aggregator as a TypedColumn that can be used in Dataset. operations.

Returns:

(undocumented)

Since:

1.6.0