org.apache.spark.scheduler

Class JobLogger

Object

org.apache.spark.scheduler.JobLogger

All Implemented Interfaces:

Logging, SparkListener

public class JobLogger
extends Object
implements SparkListener, Logging

:: DeveloperApi :: A logger class to record runtime information for jobs in Spark. This class outputs one log file for each Spark job, containing tasks start/stop and shuffle information. JobLogger is a subclass of SparkListener, use addSparkListener to add JobLogger to a SparkContext after the SparkContext is created. Note that each JobLogger only works for one SparkContext

NOTE: The functionality of this class is heavily stripped down to accommodate for a general refactor of the SparkListener interface. In its place, the EventLoggingListener is introduced to log application information as SparkListenerEvents. To enable this functionality, set spark.eventLog.enabled to true.

Constructor Summary

Constructors

Constructor and Description
JobLogger()
JobLogger(String user, String logDirName)

Method Summary

Methods

Modifier and Type	Method and Description
String	logDirName()
void	onJobEnd(SparkListenerJobEnd jobEnd) When job ends, recording job completion status and close log file
void	onJobStart(SparkListenerJobStart jobStart) When job starts, record job property and stage graph
void	onStageCompleted(SparkListenerStageCompleted stageCompleted) When stage is completed, record stage completion status
void	onStageSubmitted(SparkListenerStageSubmitted stageSubmitted) When stage is submitted, record stage submit info
void	onTaskEnd(SparkListenerTaskEnd taskEnd) When task ends, record task completion status and metrics
String	user()

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.scheduler.SparkListener

onApplicationEnd, onApplicationStart, onBlockManagerAdded, onBlockManagerRemoved, onBlockUpdated, onEnvironmentUpdate, onExecutorAdded, onExecutorMetricsUpdate, onExecutorRemoved, onTaskGettingResult, onTaskStart, onUnpersistRDD

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

JobLogger

public JobLogger(String user,
         String logDirName)

JobLogger

public JobLogger()

Method Detail

user

public String user()

logDirName

public String logDirName()

onStageSubmitted

public void onStageSubmitted(SparkListenerStageSubmitted stageSubmitted)

When stage is submitted, record stage submit info

Specified by:

onStageSubmitted in interface SparkListener

Parameters:

stageSubmitted - Stage submitted event

onStageCompleted

public void onStageCompleted(SparkListenerStageCompleted stageCompleted)

When stage is completed, record stage completion status

Specified by:

onStageCompleted in interface SparkListener

Parameters:

stageCompleted - Stage completed event

onTaskEnd

public void onTaskEnd(SparkListenerTaskEnd taskEnd)

When task ends, record task completion status and metrics

Specified by:

onTaskEnd in interface SparkListener

Parameters:

taskEnd - Task end event

onJobEnd

public void onJobEnd(SparkListenerJobEnd jobEnd)

When job ends, recording job completion status and close log file

Specified by:

onJobEnd in interface SparkListener

Parameters:

jobEnd - Job end event

onJobStart

public void onJobStart(SparkListenerJobStart jobStart)

When job starts, record job property and stage graph

Specified by:

onJobStart in interface SparkListener

Parameters:

jobStart - Job start event