How to set up logging level for Spark application in IntelliJ IDEA?

Put your log4j.properties file under a directory marked as resources, spark will read this log4j configuration.

Setting the log level on the SparkContext worked for me under Eclipse

spark.sparkContext.setLogLevel("WARN")  

If you are working on the local development with IDE, you can change the log level at run-time by:

LogManager.getRootLogger.setLevel(Level.ALL)

Ps: Put that line after the SparkContext/ SQLContext was created in your code.

I would love to figure out how to do this with a project local properties file (an example file would be nice), but I was able to get this done in Spark 2.2 with the following code:

import org.apache.log4j.{Level, Logger}

object MySparkApp {

    def main(args: Array[String]): Unit = {
      Logger.getLogger("org.apache.spark").setLevel(Level.WARN)