Convert scala list to DataFrame or DataSet
Use Seq:
val spark = SparkSession.builder().appName("Spark-SQL").master("local[2]").getOrCreate()
import spark.implicits._
var tom = new TestPerson("Tom Hanks",37,35.5)
var sam = new TestPerson("Sam Smith",40,40.5)
val PersonList = mutable.MutableList[TestPerson]()
//Adding data in list
PersonList += tom
PersonList += sam
//It will be work.
var personDS = Seq(PersonList).toDS()
SQLContext.implicits
case class TestPerson(name: String, age: Long, salary: Double)
val spark = SparkSession.builder().appName("List to Dataset").master("local[*]").getOrCreate()
var tom = new TestPerson("Tom Hanks",37,35.5)
var sam = new TestPerson("Sam Smith",40,40.5)
// mutable.MutableList[TestPerson]() is not required , i used below way which was
// cleaner
val PersonList = List(tom,sam)
import spark.implicits._
PersonList.toDS().show
Try without Seq
:
case class TestPerson(name: String, age: Long, salary: Double)
val tom = TestPerson("Tom Hanks",37,35.5)
val sam = TestPerson("Sam Smith",40,40.5)
val PersonList = mutable.MutableList[TestPerson]()
PersonList += tom
PersonList += sam
val personDS = PersonList.toDS()
println(personDS.getClass)
personDS.show()
val personDF = PersonList.toDF()
println(personDF.getClass)
personDF.show()
personDF.select("name", "age").show()
Output:
class org.apache.spark.sql.Dataset
+---------+---+------+
| name|age|salary|
+---------+---+------+
|Tom Hanks| 37| 35.5|
|Sam Smith| 40| 40.5|
+---------+---+------+
class org.apache.spark.sql.DataFrame
+---------+---+------+
| name|age|salary|
+---------+---+------+
|Tom Hanks| 37| 35.5|
|Sam Smith| 40| 40.5|
+---------+---+------+
+---------+---+
| name|age|
+---------+---+
|Tom Hanks| 37|
|Sam Smith| 40|
+---------+---+
Also, make sure to move the declaration of the case class TestPerson
outside the scope of your object.