The data type for collections of multiple values.
The data type for collections of multiple values.
Internally these are represented as columns that contain a
.scala.collection.Seq
Please use DataTypes.createArrayType()
to create a specific instance.
An ArrayType object comprises two fields, elementType: DataType
and
containsNull: Boolean
. The field of elementType
is used to specify the type of
array elements. The field of containsNull
is used to specify if the array has null
values.
The data type of values.
Indicates if values have null
values
1.3.0
An internal type used to represent everything that is not null, UDTs, arrays, structs, and maps.
An internal type used to represent everything that is not null, UDTs, arrays, structs, and maps.
The data type representing Array[Byte]
values.
The data type representing Array[Byte]
values.
Please use the singleton DataTypes.BinaryType
.
The data type representing Boolean
values.
The data type representing Boolean
values. Please use the singleton DataTypes.BooleanType
.
1.3.0
The data type representing Byte
values.
The data type representing Byte
values. Please use the singleton DataTypes.ByteType
.
1.3.0
The data type representing calendar time intervals.
The data type representing calendar time intervals. The calendar time interval is stored internally in two components: number of months the number of microseconds.
Please use the singleton DataTypes.CalendarIntervalType
.
1.5.0
Calendar intervals are not comparable.
Hive char type.
Hive char type. Similar to other HiveStringType's, these datatypes should only used for parsing, and should NOT be used anywhere else. Any instance of these data types should be replaced by a StringType before analysis.
The base type of all Spark SQL data types.
The base type of all Spark SQL data types.
1.3.0
A date type, supporting "0001-01-01" through "9999-12-31".
A date type, supporting "0001-01-01" through "9999-12-31".
Please use the singleton DataTypes.DateType
.
Internally, this is represented as the number of days from 1970-01-01.
1.3.0
A mutable implementation of BigDecimal that can hold a Long if values are small enough.
A mutable implementation of BigDecimal that can hold a Long if values are small enough.
The semantics of the fields are as follows: - _precision and _scale represent the SQL precision and scale we are looking for - If decimalVal is set, it represents the whole decimal value - Otherwise, the decimal value is longVal / (10 ** _scale)
The data type representing java.math.BigDecimal
values.
The data type representing java.math.BigDecimal
values.
A Decimal that must have fixed precision (the maximum number of digits) and scale (the number
of digits on right side of dot).
The precision can be up to 38, scale can also be up to 38 (less or equal to precision).
The default precision and scale is (10, 0).
Please use DataTypes.createDecimalType()
to create a specific instance.
1.3.0
The data type representing Double
values.
The data type representing Double
values. Please use the singleton DataTypes.DoubleType
.
1.3.0
The data type representing Float
values.
The data type representing Float
values. Please use the singleton DataTypes.FloatType
.
1.3.0
A hive string type for compatibility.
A hive string type for compatibility. These datatypes should only used for parsing, and should NOT be used anywhere else. Any instance of these data types should be replaced by a StringType before analysis.
The data type representing Int
values.
The data type representing Int
values. Please use the singleton DataTypes.IntegerType
.
1.3.0
The data type representing Long
values.
The data type representing Long
values. Please use the singleton DataTypes.LongType
.
1.3.0
The data type for Maps.
The data type for Maps. Keys in a map are not allowed to have null
values.
Please use DataTypes.createMapType()
to create a specific instance.
The data type of map keys.
The data type of map values.
Indicates if map values have null
values.
Metadata is a wrapper over Map[String, Any] that limits the value type to simple ones: Boolean, Long, Double, String, Metadata, Array[Boolean], Array[Long], Array[Double], Array[String], and Array[Metadata].
Metadata is a wrapper over Map[String, Any] that limits the value type to simple ones: Boolean, Long, Double, String, Metadata, Array[Boolean], Array[Long], Array[Double], Array[String], and Array[Metadata]. JSON is used for serialization.
The default constructor is private. User should use either MetadataBuilder or
Metadata.fromJson()
to create Metadata instances.
1.3.0
Builder for Metadata.
Builder for Metadata. If there is a key collision, the latter will overwrite the former.
1.3.0
The data type representing NULL
values.
The data type representing NULL
values. Please use the singleton DataTypes.NullType
.
1.3.0
Numeric data types.
Numeric data types.
1.3.0
Represents a JVM object that is passing through Spark SQL expression evaluation.
Represents a JVM object that is passing through Spark SQL expression evaluation.
The data type representing Short
values.
The data type representing Short
values. Please use the singleton DataTypes.ShortType
.
1.3.0
The data type representing String
values.
The data type representing String
values. Please use the singleton DataTypes.StringType
.
1.3.0
A field inside a StructType.
A field inside a StructType.
The name of this field.
The data type of this field.
Indicates if values of this field can be null
values.
The metadata of this field. The metadata should be preserved during transformation if the content of the column is not modified, e.g, in selection.
1.3.0
A StructType object can be constructed by
A StructType object can be constructed by
StructType(fields: Seq[StructField])
For a StructType object, one or multiple StructFields can be extracted by names.
If multiple StructFields are extracted, a StructType object will be returned.
If a provided name does not have a matching field, it will be ignored. For the case
of extracting a single StructField, a null
will be returned.
Scala Example:
import org.apache.spark.sql._ import org.apache.spark.sql.types._ val struct = StructType( StructField("a", IntegerType, true) :: StructField("b", LongType, false) :: StructField("c", BooleanType, false) :: Nil) // Extract a single StructField. val singleField = struct("b") // singleField: StructField = StructField(b,LongType,false) // If this struct does not have a field called "d", it throws an exception. struct("d") // java.lang.IllegalArgumentException: Field "d" does not exist. // ... // Extract multiple StructFields. Field names are provided in a set. // A StructType object will be returned. val twoFields = struct(Set("b", "c")) // twoFields: StructType = // StructType(StructField(b,LongType,false), StructField(c,BooleanType,false)) // Any names without matching fields will throw an exception. // For the case shown below, an exception is thrown due to "d". struct(Set("b", "c", "d")) // java.lang.IllegalArgumentException: Field "d" does not exist. // ...
A org.apache.spark.sql.Row object is used as a value of the StructType.
Scala Example:
import org.apache.spark.sql._ import org.apache.spark.sql.types._ val innerStruct = StructType( StructField("f1", IntegerType, true) :: StructField("f2", LongType, false) :: StructField("f3", BooleanType, false) :: Nil) val struct = StructType( StructField("a", innerStruct, true) :: Nil) // Create a Row with the schema defined by struct val row = Row(Row(1, 2, true))
1.3.0
The data type representing java.sql.Timestamp
values.
The data type representing java.sql.Timestamp
values.
Please use the singleton DataTypes.TimestampType
.
1.3.0
Hive varchar type.
Hive varchar type. Similar to other HiveStringType's, these datatypes should only used for parsing, and should NOT be used anywhere else. Any instance of these data types should be replaced by a StringType before analysis.
An AbstractDataType
that matches any concrete data types.
An AbstractDataType
that matches any concrete data types.
Companion object for ArrayType.
Companion object for ArrayType.
1.3.0
1.3.0
1.3.0
1.3.0
1.5.0
1.3.0
1.3.0
Extra factory methods and pattern matchers for Decimals.
Extra factory methods and pattern matchers for Decimals.
1.3.0
1.3.0
1.3.0
Metadata key used to store the raw hive type string in the metadata of StructField.
Metadata key used to store the raw hive type string in the metadata of StructField. This is relevant for datatypes that do not have a direct Spark SQL counterpart, such as CHAR and VARCHAR. We need to preserve the original type in order to invoke the correct object inspector in Hive.
1.3.0
1.3.0
1.3.0
1.3.0
1.3.0
1.3.0
1.3.0
1.3.0
1.3.0
Contains a type system for attributes produced by relations, including complex types like structs, arrays and maps.