Encode an ADT / sealed trait hierarchy into Spark DataSet column

I have once dived deeply into the subject and created a repo showcasing all the approaches I have found could be useful.

Link: https://github.com/atais/spark-enum

Generally, zero323 is right, but you might find it useful to understand the full picture.

TL;DR There is no good solution right now, and given Spark SQL / Dataset implementation, it is unlikely there will be one in the foreseeable future.

You can use generic kryo or java encoder

val occupation: Seq[Occupation] = Seq(SoftwareEngineer, Wizard(1), Other("foo"))
spark.createDataset(occupation)(org.apache.spark.sql.Encoders.kryo[Occupation])

but is hardly useful in practice.

UDT API provides another possible approach as for now (Spark 1.6, 2.0, 2.1-SNAPSHOT) it is private and requires quite a lot boilerplate code (you can check o.a.s.ml.linalg.VectorUDT to see example implementation).