Spark not using spark.sql.parquet.compression.codec

Worked for me in 2.1.1

df.write.option("compression","snappy").parquet(filename)

For spark 1.3 and spark.sql.parquet.compression.codec parameter did not compress the output. But the below one did work.

sqlContext.sql("SET parquet.compression=SNAPPY")

Try this. Seems to work for me in 1.6.0

val sc = new SparkContext(sparkConf)
val sqlContext = new HiveContext(sc)
sqlContext.setConf("spark.sql.parquet.compression.codec", "uncompressed")

Related