PySpark - rename more than one column using withColumnRenamed
It is not possible to use a single withColumnRenamed
call.
You can use
DataFrame.toDF
method*data.toDF('x3', 'x4')
or
new_names = ['x3', 'x4'] data.toDF(*new_names)
It is also possible to rename with simple
select
:from pyspark.sql.functions import col mapping = dict(zip(['x1', 'x2'], ['x3', 'x4'])) data.select([col(c).alias(mapping.get(c, c)) for c in data.columns])
Similarly in Scala you can:
Rename all columns:
val newNames = Seq("x3", "x4") data.toDF(newNames: _*)
Rename from mapping with
select
:val mapping = Map("x1" -> "x3", "x2" -> "x4") df.select( df.columns.map(c => df(c).alias(mapping.get(c).getOrElse(c))): _* )
or
foldLeft
+withColumnRenamed
mapping.foldLeft(data){ case (data, (oldName, newName)) => data.withColumnRenamed(oldName, newName) }
* Not to be confused with RDD.toDF
which is not a variadic functions, and takes column names as a list,
I couldn't find an easy pyspark solution either, so just built my own one, similar to pandas' df.rename(columns={'old_name_1':'new_name_1', 'old_name_2':'new_name_2'})
.
import pyspark.sql.functions as F
def rename_columns(df, columns):
if isinstance(columns, dict):
return df.select(*[F.col(col_name).alias(columns.get(col_name, col_name)) for col_name in df.columns])
else:
raise ValueError("'columns' should be a dict, like {'old_name_1':'new_name_1', 'old_name_2':'new_name_2'}")
So your solution will look like data = rename_columns(data, {'x1': 'x3', 'x2': 'x4'})
If you want to chain your method calls, Spark 3.0 brought in pyspark.sql.DataFrame.transform, which you can use in the following way:
my_df.transform(lambda df: rename_columns(df, {'old_name_1':'new_name_1', 'old_name_2':'new_name_2'}))
It saves me some lines of code, hope it will help you too.
why do you want to perform it in a single line if you print the execution plan it is actually done in single line only
data = spark.createDataFrame([(1,2), (3,4)], ['x1', 'x2'])
data = (data
.withColumnRenamed('x1','x3')
.withColumnRenamed('x2', 'x4'))
data.explain()
OUTPUT
== Physical Plan ==
*(1) Project [x1#1548L AS x3#1552L, x2#1549L AS x4#1555L]
+- Scan ExistingRDD[x1#1548L,x2#1549L]
if you want to do it with a tuple of list you can use a simple map function
data = spark.createDataFrame([(1,2), (3,4)], ['x1', 'x2'])
new_names = [("x1","x3"),("x2","x4")]
data = data.select(list(
map(lambda old,new:F.col(old).alias(new),*zip(*new_names))
))
data.explain()
still has same plan
OUTPUT
== Physical Plan ==
*(1) Project [x1#1650L AS x3#1654L, x2#1651L AS x4#1655L]
+- Scan ExistingRDD[x1#1650L,x2#1651L]