PySpark fix/remove console progress bar

You could either disable by setting

• spark.ui.showConsoleProgress = False

or

• decrease logging level in log4j.properties higher than INFO, i.e. to ERROR

Relevant Spark jiras:

• https://issues.apache.org/jira/browse/SPARK-4017
• https://issues.apache.org/jira/browse/SPARK-18719

spark.ui.showConsoleProgress was always in Spark, since version 1.2, but will be documented only in Spark 2.2.

Example code:

spark.conf.set('spark.ui.showConsoleProgress', False)

The answer of Tagar didn't work for me in pyspark.

Here is the workaround I found to remove progress bars from the console:

from pyspark import SparkContext, SparkConf
from pyspark.sql.session import SparkSession


conf = SparkConf().set("spark.ui.showConsoleProgress", "false")
sc = SparkContext(appName="RandomForest", conf=conf)
spark = SparkSession(sc)

Hope this helps!