Spark UDF error - Schema for type Any is not supported

It's because of the else returning a String: "avg_acc_x". Take away the quotes:

val noNegative = udf {(avg_acc_x: Double) => if(avg_acc_x < 0) 0 else avg_acc_x}

This error basically occurs when the Analyzer cannot resolve the type of udf properly. That is, the types are mixed (Int and String).

Therefore,

if(avg_acc_x < 0) 0 else avg_acc_x (Integer Type)

if(avg_acc_x < 0) "0" else "avg_acc_x" (String Type)

should resolve this exception.

Related