pyspark dataframe join drop duplicate columns code example

Example 1: remove duplicates based on two columns in dataframe

df.drop_duplicates(['A','B'],keep= 'last')

Example 2: remove duplicate columns python dataframe

df = df.loc[:,~df.columns.duplicated()]