Fetch Spark dataframe column list

You can use the following command:

val selectColumns = dataset1.columns.toSeq

---

scala> val dataset1 = Seq(("66", "a", "4"), ("67", "a", "0"), ("70", "b", "4"), ("71", "d", "4")).toDF("KEY1", "KEY2", "ID")
dataset1: org.apache.spark.sql.DataFrame = [KEY1: string, KEY2: string ... 1 more field]

scala> val selectColumns = dataset1.columns.toSeq
selectColumns: Seq[String] = WrappedArray(KEY1, KEY2, ID)

val selectColumns = dataset1.columns.toList.map(col(_))

I use the columns property like so

val cols = dataset1.columns.toSeq

and then if you are selecting all the columns later on in the order of the Sequence from head to tail you can use

val orderedDF = dataset1.select(cols.head, cols.tail:_ *)

we can get the column names of a dataset / table into a Sequence variable in following ways.

from Dataset,

val col_seq:Seq[String] = dataset.columns.toSeq

from table,

val col_seq:Seq[String] = spark.table("tablename").columns.toSeq
                           or
val col_seq:Seq[String] = spark.catalog.listColumns("tablename").select('name).collect.map(col=>col.toString).toSeq