Filter dataframe on non-empty WrappedArray
I don't think you need to use a UDF here.
You can just use size
method and filter all those rows with array size = 0
df.filter(""" size(fahrspur_liste) != 0 """)
You can do this with an udf
in Spark:
val removeEmpty = udf((array: Seq[Long]) => !array.isEmpty)
val df2 = df.filter(removeEmpty($"fahrspur_liste"))
Here the udf
checks if the array is empty or not. The filter function will then remove those that come back as true.