How to suppress Spark logging in unit tests?

In my case one of my own libraries brought logback-classic into the mix. This materialized in a warning at the start:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/alex/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.1.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/alex/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

I solved this by excluding it from the dependency:

"com.mystuff" % "mylib" % "1.0.0" exclude("ch.qos.logback", "logback-classic")

Now I could add a log4j.properties file in test/resources which now gets used by Spark.

Add the following code into the log4j.properties file inside the src/test/resources dir, create the file/dir if not exist

# Change this to set Spark log level
log4j.logger.org.apache.spark=WARN

# Silence akka remoting
log4j.logger.Remoting=WARN

# Ignore messages below warning level from Jetty, because it's a bit verbose
log4j.logger.org.eclipse.jetty=WARN

When I run my unit tests (I'm using JUnit and Maven), I only receive WARN level logs, in other words no more cluttering with INFO level logs (though they can be useful at times for debugging).

I hope this helps.

After some time of struggling with Spark log output as well, I found a blog post with a solution I particularly liked.

If you use slf4j, one can simply exchange the underlying log implementation. A good canidate for the test scope is slf4j-nop, which carfully takes the log output and puts it where the sun never shines.

When using Maven you can add the following to the top of your dependencies list:

<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-api</artifactId>
  <version>1.7.12</version>
  <scope>provided</scope>
</dependency>

<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-nop</artifactId>
  <version>1.7.12</version>
  <scope>test</scope>
</dependency>

Note that it might be important to have it at the beginning of the dependencies list to make sure that the given implementations are used instead of those that might come with other packages (and which you can consider to exclude in order to keep your class path tidy and avoid unexpected conflicts).

You can use a separate Logback config for tests. Depending on your environment it's possible that you just need to create conf/logback-test.xml with something that hides the logs. I think this should do that:

<configuration>
  <root level="debug">
  </root>
</configuration>

As I understand it, this captures all logs (level debug and higher) and assigns no logger to them, so they get discarded. A better option is to configure a file logger for them, so you can still access the logs if you want to.

See http://logback.qos.ch/manual/configuration.html for the detailed documentation.