Spark2.1.0 incompatible Jackson versions 2.7.6
Solution with Gradle, by using the resolutionStrategy (https://docs.gradle.org/current/dsl/org.gradle.api.artifacts.ResolutionStrategy.html):
configurations {
all {
resolutionStrategy {
force 'com.fasterxml.jackson.core:jackson-core:2.4.4', 'com.fasterxml.jackson.core:jackson-databind:2.4.4', 'com.fasterxml.jackson.core:jackson-annotations:2.4.4'
}
}
}
Spark 2.1.0 contains com.fasterxml.jackson.core
as transitive dependency. So, we do not need to include then in libraryDependencies
.
But if you want to add a different com.fasterxml.jackson.core
dependencies' version then you have to override them. Like this:
name := "testSpark2"
version := "1.0"
scalaVersion := "2.11.8"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7"
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.7"
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-mllib_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-repl_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-streaming-flume_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-network-shuffle_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-hive_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-streaming-flume-assembly_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-mesos_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-graphx_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-catalyst_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-launcher_2.11" % "2.1.0"
So, change your build.sbt
like the one above and it will work as expected.
I hope it helps!
FYI. For my case, I'm using spark and kafka-streams
in the app, while kafka-streams uses com.fasterxml.jackson.core 2.8.5. Adding exclude
as below fixed the issue
(gradle)
compile (group: "org.apache.kafka", name: "kafka-streams", version: "0.11.0.0"){
exclude group:"com.fasterxml.jackson.core"
}