is Parquet predicate pushdown works on S3 using Spark non EMR?

I was wondering this myself so I just tested it out. We use EMR clusters and Spark 1.6.1 .

• I generated some dummy data in Spark and saved it as a parquet file locally as well as on S3.
• I created multiple Spark jobs with different kind of filters and column selections. I ran these tests once for the local file and once for the S3 file.
• I then used the Spark History Server to see how much data each job had as input.

Results:

• For the local parquet file: The results showed that the column selection and filters were pushed down to the read as the input size was reduced when the job contained filters or column selection.
• For the S3 parquet file: The input size was always the same as the Spark job that processed all of the data. None of the filters or column selections were pushed down to the read. The parquet file was always completely loaded from S3. Even though the query plan (.queryExecution.executedPlan) showed that the filters were pushed down.

I will add more details about the tests and results when I have time.

Yes. Filter pushdown does not depend on the underlying file system. It only depends on the spark.sql.parquet.filterPushdown and the type of filter (not all filters can be pushed down).

See https://github.com/apache/spark/blob/v2.2.0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L313 for the pushdown logic.