org.apache.spark.util

Class FileLogger

Object

org.apache.spark.util.FileLogger

All Implemented Interfaces:

Logging

public class FileLogger
extends Object
implements Logging

A generic class for logging information to file.

Constructor Summary

Constructors

Constructor and Description
FileLogger(String logDir, SparkConf sparkConf)
FileLogger(String logDir, SparkConf sparkConf, boolean compress)
FileLogger(String logDir, SparkConf sparkConf, boolean compress, boolean overwrite)
FileLogger(String logDir, SparkConf sparkConf, org.apache.hadoop.conf.Configuration hadoopConf, int outputBufferSize, boolean compress, boolean overwrite, scala.Option<org.apache.hadoop.fs.permission.FsPermission> dirPermissions)

Method Summary

Methods

Modifier and Type	Method and Description
void	close() Close the writer.
int	fileIndex()
void	flush() Flush the writer to disk manually.
void	log(String msg, boolean withTime) Log the message to the given writer.
void	logLine(String msg, boolean withTime) Log the message to the given writer as a new line.
void	newFile(String fileName, scala.Option<org.apache.hadoop.fs.permission.FsPermission> perms) Start a writer for a new file, closing the existing one if it exists.
void	start() Start this logger by creating the logging directory.
void	stop() Close all open writers, streams, and file systems.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

FileLogger

public FileLogger(String logDir,
          SparkConf sparkConf,
          org.apache.hadoop.conf.Configuration hadoopConf,
          int outputBufferSize,
          boolean compress,
          boolean overwrite,
          scala.Option<org.apache.hadoop.fs.permission.FsPermission> dirPermissions)

FileLogger

public FileLogger(String logDir,
          SparkConf sparkConf,
          boolean compress,
          boolean overwrite)

FileLogger

public FileLogger(String logDir,
          SparkConf sparkConf,
          boolean compress)

FileLogger

public FileLogger(String logDir,
          SparkConf sparkConf)

Method Detail

fileIndex

public int fileIndex()

start

public void start()

Start this logger by creating the logging directory.

log

public void log(String msg,
       boolean withTime)

Log the message to the given writer.

Parameters:

msg - The message to be logged

withTime - Whether to prepend message with a timestamp

logLine

public void logLine(String msg,
           boolean withTime)

Log the message to the given writer as a new line.

Parameters:

msg - The message to be logged

withTime - Whether to prepend message with a timestamp

flush

public void flush()

Flush the writer to disk manually.

When using a Hadoop filesystem, we need to invoke the hflush or sync method. In HDFS, hflush guarantees that the data gets to all the DataNodes.

close

public void close()

Close the writer. Any subsequent calls to log or flush will have no effect.

newFile

public void newFile(String fileName,
           scala.Option<org.apache.hadoop.fs.permission.FsPermission> perms)

Start a writer for a new file, closing the existing one if it exists.

Parameters:

fileName - Name of the new file, defaulting to the file index if not provided.

perms - Permissions to put on the new file.

stop

public void stop()

Close all open writers, streams, and file systems. Any subsequent uses of this FileLogger instance will throw exceptions.