Difference between na().drop() and filter(col.isNotNull) (Apache Spark)
With df.na.drop()
you drop the rows containing any null or NaN values.
With df.filter(df.col("onlyColumnInOneColumnDataFrame").isNotNull())
you drop those rows which have null only in the column onlyColumnInOneColumnDataFrame
.
If you would want to achieve the same thing, that would be df.na.drop(["onlyColumnInOneColumnDataFrame"])
.