Passing Array to Python Spark Lit Function
for loop in array inbuilt function
You can use array
inbuilt function as
a = [1,2,3,4,5,6,7,8,9,10]
df = spark.createDataFrame([['a b c d e f g h i j '],], ['col1'])
df = df.withColumn("NewColumn", F.array([F.lit(x) for x in a]))
df.show(truncate=False)
You should get
+--------------------+-------------------------------+
|col1 |NewColumn |
+--------------------+-------------------------------+
|a b c d e f g h i j |[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]|
+--------------------+-------------------------------+
root
|-- col1: string (nullable = true)
|-- NewColumn: array (nullable = false)
| |-- element: integer (containsNull = false)
Using udf function
#udf function
def arrayUdf():
return a
callArrayUdf = F.udf(arrayUdf, T.ArrayType(T.IntegerType()))
#calling udf function
df = df.withColumn("NewColumn", callArrayUdf())
output is same as with for loop way
Updated
I am pasting @pault's comment given below
You can hide the loop using
map
:df.withColumn("NewColumn", F.array(map(F.lit, a)))