How to suppress the "Stage 2===>" from the output console in spark?
You need to set spark.ui.showConsoleProgress
to false
I found this in the comments of the ticket for the addition of the progress bar.
https://issues.apache.org/jira/browse/SPARK-4017
I haven't seen it in any of the documentation though, it really should be added.
If you want to do it by code. Add the following when you are creating the SparkContext:
import org.apache.log4j.{Level, Logger}
import org.apache.spark.{SparkConf, SparkContext}
Logger.getRootLogger.setLevel(Level.ERROR) // Disabling "INFO" level logs (these lines must be before to create the SparkContext)
val conf = new SparkConf().set("spark.ui.showConsoleProgress", "false").setAppName("myApp")
val sc = new SparkContext(conf)
UPDATE CONTENT FOR SPARK2+:
Using SparkSession you can supress those messages adding the following line (.config("spark.ui.showConsoleProgress", "false")
) to the declaration:
spark = SparkSession
.builder
.master("local[*]")
.appName("myApp")
.config("spark.ui.showConsoleProgress", "false")