How to create a Row from a List or Array in Spark using Scala
Something like the following should work:
import org.apache.spark.sql._
def f(n: List[Int], s: String) : Row =
Row.fromSeq(s.split(",").zipWithIndex.collect{case (a,b) if n.contains(b) => a}.toSeq)
You are missing creation of the StructField and StructType. Refer to the official guide http://spark.apache.org/docs/latest/sql-programming-guide.html, part Programmatically Specifying the Schema
I'm not a Scala specialist, but in Python it would look like this:
from pyspark.sql import *
sqlContext = SQLContext(sc)
input = [1,2]
def parse(line):
global input
l = line.split(',')
res = [l[0]]
for ind in input:
res.append(l[ind])
return res
csv = sc.textFile("file:///tmp/inputfile.csv")
rows = csv.map(lambda x: parse(x))
fieldnum = len(input) + 1
fields = [StructField("col"+str(i), StringType(), True) for i in range(fieldnum)]
schema = StructType(fields)
csvWithSchema = sqlContext.applySchema(rows, schema)
csvWithSchema.registerTempTable("test")
sqlContext.sql("SELECT * FROM test").collect()
In short, you should not directly convert them to Row objects, just leave as RDD and apply schema to it with applySchema
You can also try:
Row.fromSeq(line(0).toString ++ line(1).toDouble ++ line(2).toDouble ++ line.slice(2, line.size).map(value => value.toString))