Spark unable to download kafka library

First: As discussed on Developers Mailing list, Kafka is not included in binary distribution. That is why you don't have it on classpath.

Second: in your --packages command, you should specify Scala version. It's not necessary only in SBT, but spark-submit uses Ivy in the background.

So, please try:

  $ bin/spark-submit --packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.2.0 script.py

Extra point: Maybe I will create a PR to change description, it's misleading

Try to write

bin/spark-submit --jars yourjarfile.jar --packages org.apache.spark:spark-streaming-kafka-0-8-assembly_2.11:2.4.3 pythoncode.py

I had the same problem and I solved it typing like this. I hope that helps.