org.apache.spark.input

Class PortableDataStream

Object

org.apache.spark.input.PortableDataStream

All Implemented Interfaces:

java.io.Serializable

public class PortableDataStream
extends Object
implements scala.Serializable

A class that allows DataStreams to be serialized and moved around by not creating them until they need to be read

See Also:

Serialized Form

Note:

TaskAttemptContext is not serializable resulting in the confBytes construct, CombineFileSplit is not serializable resulting in the splitBytes construct

Constructor Summary

Constructors

Constructor and Description
PortableDataStream(org.apache.hadoop.mapreduce.lib.input.CombineFileSplit isplit, org.apache.hadoop.mapreduce.TaskAttemptContext context, Integer index)

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.conf.Configuration	getConfiguration()
String	getPath()
java.io.DataInputStream	open() Create a new DataInputStream from the split and context.
byte[]	toArray() Read the file as a byte array

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PortableDataStream

public PortableDataStream(org.apache.hadoop.mapreduce.lib.input.CombineFileSplit isplit,
                          org.apache.hadoop.mapreduce.TaskAttemptContext context,
                          Integer index)

Method Detail

open

public java.io.DataInputStream open()

Create a new DataInputStream from the split and context. The user of this method is responsible for closing the stream after usage.

Returns:

(undocumented)

toArray

public byte[] toArray()

Read the file as a byte array

Returns:

(undocumented)

getPath

public String getPath()

getConfiguration

public org.apache.hadoop.conf.Configuration getConfiguration()