How to suppress the "Stage 2===>" from the output console in spark?

You need to set spark.ui.showConsoleProgress to false

I found this in the comments of the ticket for the addition of the progress bar.

https://issues.apache.org/jira/browse/SPARK-4017

I haven't seen it in any of the documentation though, it really should be added.


If you want to do it by code. Add the following when you are creating the SparkContext:

import org.apache.log4j.{Level, Logger}
import org.apache.spark.{SparkConf, SparkContext}

Logger.getRootLogger.setLevel(Level.ERROR)  // Disabling "INFO" level logs (these lines must be before to create the SparkContext)
val conf = new SparkConf().set("spark.ui.showConsoleProgress", "false").setAppName("myApp")  
val sc = new SparkContext(conf)

UPDATE CONTENT FOR SPARK2+:

Using SparkSession you can supress those messages adding the following line (.config("spark.ui.showConsoleProgress", "false")) to the declaration:

spark = SparkSession
      .builder
      .master("local[*]")
      .appName("myApp")
      .config("spark.ui.showConsoleProgress", "false")