pyspark.SparkContext.setInterruptOnCancel

SparkContext.setInterruptOnCancel(interruptOnCancel)

Set the behavior of job cancellation from jobs started in this thread.

New in version 3.5.0.

Parameters

interruptOnCancelbool

If true, then job cancellation will result in Thread.interrupt() being called on the job’s executor threads. This is useful to help ensure that the tasks are actually stopped in a timely manner, but is off by default due to HDFS-1208, where HDFS may respond to Thread.interrupt() by marking nodes as dead.

See also

SparkContext.addJobTag()

SparkContext.removeJobTag()

SparkContext.cancelAllJobs()

SparkContext.cancelJobGroup()

SparkContext.cancelJobsWithTag()