How to make shark/spark clear the cache?

Are you using the cache() method to persist RDDs?

cache() just calls persist(), so to remove the cache for an RDD, call unpersist().

To remove all cached data:

sqlContext.clearCache()

Source: https://spark.apache.org/docs/2.0.1/api/java/org/apache/spark/sql/SQLContext.html

If you want to remove an specific Dataframe from cache:

df.unpersist()

Related