Spark - Group by HAVING with dataframe syntax?

Yes, it doesn't exist. You express the same logic with agg followed by where:

df.groupBy(someExpr).agg(somAgg).where(somePredicate) 

Say for example if I want to find products in each category, having fees less than 3200 and their count must not be less than 10:

  • SQL query:
sqlContext.sql("select Category,count(*) as 
count from hadoopexam where HadoopExamFee<3200  
group by Category having count>10")
  • DataFrames API (Pyspark)
from pyspark.sql.functions import *

df.filter(df.HadoopExamFee<3200)
  .groupBy('Category')
  .agg(count('Category').alias('count'))
  .filter(col('count')>10)