scheduler

Type Members

1. case class BatchInfo(batchTime: Time, streamIdToInputInfo: Map[Int, StreamInputInfo], submissionTime: Long, processingStartTime: Option[Long], processingEndTime: Option[Long], outputOperationInfos: Map[Int, OutputOperationInfo]) extends Product with Serializable

   

   :: DeveloperApi :: Class having information on completed batches.

   :: DeveloperApi :: Class having information on completed batches.

   batchTime

   Time of the batch

   streamIdToInputInfo

   A map of input stream id to its input info

   submissionTime

   Clock time of when jobs of this batch was submitted to the streaming scheduler queue

   processingStartTime

   Clock time of when the first job of this batch started processing

   processingEndTime

   Clock time of when the last job of this batch finished processing

   outputOperationInfos

   The output operations in this batch

   Annotations

   @DeveloperApi()

2. case class OutputOperationInfo(batchTime: Time, id: Int, name: String, description: String, startTime: Option[Long], endTime: Option[Long], failureReason: Option[String]) extends Product with Serializable

   

   :: DeveloperApi :: Class having information on output operations.

   :: DeveloperApi :: Class having information on output operations.

   batchTime

   Time of the batch

   id

   Id of this output operation. Different output operations have different ids in a batch.

   name

   The name of this output operation.

   description

   The description of this output operation.

   startTime

   Clock time of when the output operation started processing

   endTime

   Clock time of when the output operation started processing

   failureReason

   Failure reason if this output operation fails

   Annotations

   @DeveloperApi()

3. case class ReceiverInfo(streamId: Int, name: String, active: Boolean, location: String, executorId: String, lastErrorMessage: String = "", lastError: String = "", lastErrorTime: Long = 1L) extends Product with Serializable

   

   :: DeveloperApi :: Class having information about a receiver

   :: DeveloperApi :: Class having information about a receiver

   Annotations

   @DeveloperApi()

4. class StatsReportListener extends StreamingListener

   

   :: DeveloperApi :: A simple StreamingListener that logs summary statistics across Spark Streaming batches

   :: DeveloperApi :: A simple StreamingListener that logs summary statistics across Spark Streaming batches

   Annotations

   @DeveloperApi()

5. case class StreamInputInfo(inputStreamId: Int, numRecords: Long, metadata: Map[String, Any] = Map.empty) extends Product with Serializable

   

   :: DeveloperApi :: Track the information of input stream at specified batch time.

   :: DeveloperApi :: Track the information of input stream at specified batch time.

   inputStreamId

   the input stream id

   numRecords

   the number of records in a batch

   metadata

   metadata for this batch. It should contain at least one standard field named "Description" which maps to the content that will be shown in the UI.

   Annotations

   @DeveloperApi()

6. trait StreamingListener extends AnyRef

   

   :: DeveloperApi :: A listener interface for receiving information about an ongoing streaming computation.

   :: DeveloperApi :: A listener interface for receiving information about an ongoing streaming computation.

   Annotations

   @DeveloperApi()

7. case class StreamingListenerBatchCompleted(batchInfo: BatchInfo) extends StreamingListenerEvent with Product with Serializable

   

   Annotations

   @DeveloperApi()

8. case class StreamingListenerBatchStarted(batchInfo: BatchInfo) extends StreamingListenerEvent with Product with Serializable

   

   Annotations

   @DeveloperApi()

9. case class StreamingListenerBatchSubmitted(batchInfo: BatchInfo) extends StreamingListenerEvent with Product with Serializable

   

   Annotations

   @DeveloperApi()

10. sealed trait StreamingListenerEvent extends AnyRef

    

    :: DeveloperApi :: Base trait for events related to StreamingListener

    :: DeveloperApi :: Base trait for events related to StreamingListener

    Annotations

    @DeveloperApi()

11. case class StreamingListenerOutputOperationCompleted(outputOperationInfo: OutputOperationInfo) extends StreamingListenerEvent with Product with Serializable

    

    Annotations

    @DeveloperApi()

12. case class StreamingListenerOutputOperationStarted(outputOperationInfo: OutputOperationInfo) extends StreamingListenerEvent with Product with Serializable

    

    Annotations

    @DeveloperApi()

13. case class StreamingListenerReceiverError(receiverInfo: ReceiverInfo) extends StreamingListenerEvent with Product with Serializable

    

    Annotations

    @DeveloperApi()

14. case class StreamingListenerReceiverStarted(receiverInfo: ReceiverInfo) extends StreamingListenerEvent with Product with Serializable

    

    Annotations

    @DeveloperApi()

15. case class StreamingListenerReceiverStopped(receiverInfo: ReceiverInfo) extends StreamingListenerEvent with Product with Serializable

    

    Annotations

    @DeveloperApi()