How to use spark-avro package to read avro file from spark-shell?

tl;dr Since Spark 2.4.x+ provides built-in support for reading and writing Apache Avro data, but the spark-avro module is external and not included in spark-submit or spark-shell by default, you should make sure that you use the same Scala version (ex. 2.12) for the spark-shell and --packages.

---

The reason for the exception is that you use spark-shell that is from Spark built against Scala 2.11.12 while --packages specifies a dependency with Scala 2.12 (in org.apache.spark:spark-avro_2.12:2.4.0).

Use --packages org.apache.spark:spark-avro_2.11:2.4.0 and you should be fine.

just incase if some one is interested for pyspark 2.7 and spark 2.4.3

below package works

bin/pyspark --packages org.apache.spark:spark-avro_2.11:2.4.3