org.apache.spark

Class FetchFailed

Object

org.apache.spark.FetchFailed

All Implemented Interfaces:

java.io.Serializable, TaskEndReason, TaskFailedReason, scala.Equals, scala.Product

public class FetchFailed
extends Object
implements TaskFailedReason, scala.Product, scala.Serializable

:: DeveloperApi :: Task failed to fetch shuffle data from a remote node. Probably means we have lost the remote executors the task is trying to fetch from, and thus need to rerun the previous stage.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
FetchFailed(BlockManagerId bmAddress, int shuffleId, int mapId, int reduceId, String message)

Method Summary

Modifier and Type	Method and Description
BlockManagerId	bmAddress()
abstract static boolean	canEqual(Object that)
boolean	countTowardsTaskFailures() Fetch failures lead to a different failure handling path: (1) we don't abort the stage after 4 task failures, instead we immediately go back to the stage which generated the map output, and regenerate the missing data.
abstract static boolean	equals(Object that)
int	mapId()
String	message()
abstract static int	productArity()
abstract static Object	productElement(int n)
static scala.collection.Iterator<Object>	productIterator()
static String	productPrefix()
int	reduceId()
int	shuffleId()
String	toErrorString() Error message displayed in the web UI.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface scala.Product

productArity, productElement, productIterator, productPrefix

Methods inherited from interface scala.Equals

canEqual, equals

Constructor Detail

FetchFailed

public FetchFailed(BlockManagerId bmAddress,
                   int shuffleId,
                   int mapId,
                   int reduceId,
                   String message)

Method Detail

canEqual

public abstract static boolean canEqual(Object that)

equals

public abstract static boolean equals(Object that)

productElement

public abstract static Object productElement(int n)

productArity

public abstract static int productArity()

productIterator

public static scala.collection.Iterator<Object> productIterator()

productPrefix

public static String productPrefix()

bmAddress

public BlockManagerId bmAddress()

shuffleId

public int shuffleId()

mapId

public int mapId()

reduceId

public int reduceId()

message

public String message()

toErrorString

public String toErrorString()

Description copied from interface: TaskFailedReason

Error message displayed in the web UI.

Specified by:

toErrorString in interface TaskFailedReason

countTowardsTaskFailures

public boolean countTowardsTaskFailures()

Fetch failures lead to a different failure handling path: (1) we don't abort the stage after 4 task failures, instead we immediately go back to the stage which generated the map output, and regenerate the missing data. (2) we don't count fetch failures for blacklisting, since presumably its not the fault of the executor where the task ran, but the executor which stored the data. This is especially important because we might rack up a bunch of fetch-failures in rapid succession, on all nodes of the cluster, due to one bad node.

Specified by:

countTowardsTaskFailures in interface TaskFailedReason

Returns:

(undocumented)