org.apache.spark.sql.sources

HadoopFsRelationProvider

trait HadoopFsRelationProvider extends AnyRef

::Experimental:: Implemented by objects that produce relations for a specific kind of data source with a given schema and partitioned columns. When Spark SQL is given a DDL operation with a USING clause specified (to specify the implemented HadoopFsRelationProvider), a user defined schema, and an optional list of partition columns, this interface is used to pass in the parameters specified by a user.

Users may specify the fully qualified class name of a given data source. When that class is not found Spark SQL will append the class name DefaultSource to the path, allowing for less verbose invocation. For example, 'org.apache.spark.sql.json' would resolve to the data source 'org.apache.spark.sql.json.DefaultSource'

A new instance of this class will be instantiated each time a DDL call is made.

The difference between a RelationProvider and a HadoopFsRelationProvider is that users need to provide a schema and a (possibly empty) list of partition columns when using a HadoopFsRelationProvider. A relation provider can inherits both RelationProvider, and HadoopFsRelationProvider if it can support schema inference, user-specified schemas, and accessing partitioned relations.

Annotations
@Experimental()
Since

1.4.0

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. HadoopFsRelationProvider
  2. AnyRef
  3. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def createRelation(sqlContext: SQLContext, paths: Array[String], dataSchema: Option[StructType], partitionColumns: Option[StructType], parameters: Map[String, String]): HadoopFsRelation

    Returns a new base relation with the given parameters, a user defined schema, and a list of partition columns.

    Returns a new base relation with the given parameters, a user defined schema, and a list of partition columns. Note: the parameters' keywords are case insensitive and this insensitivity is enforced by the Map that is passed to the function.

    dataSchema

    Schema of data columns (i.e., columns that are not partition columns).

Concrete Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  7. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  8. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  9. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  10. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  11. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  12. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  13. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  14. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  15. final def notify(): Unit

    Definition Classes
    AnyRef

  16. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  17. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  18. def toString(): String

    Definition Classes
    AnyRef → Any

  19. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  20. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  21. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped