Spark - Group by HAVING with dataframe syntax?
Yes, it doesn't exist. You express the same logic with agg
followed by where
:
df.groupBy(someExpr).agg(somAgg).where(somePredicate)
Say for example if I want to find products in each category, having fees less than 3200 and their count must not be less than 10:
- SQL query:
sqlContext.sql("select Category,count(*) as
count from hadoopexam where HadoopExamFee<3200
group by Category having count>10")
- DataFrames API (Pyspark)
from pyspark.sql.functions import *
df.filter(df.HadoopExamFee<3200)
.groupBy('Category')
.agg(count('Category').alias('count'))
.filter(col('count')>10)