how to drop unique values in pandas dataframe code example
Example 1: pandas unique values in column
df.name.unique()
Example 2: pandas unique values to list
df.groupby('param')['column'].nunique().sort_values(ascending=False).unique().tolist()
Example 3: Return a new DataFrame with duplicate rows removed
from pyspark.sql import Row
df = sc.parallelize([
Row(name='Alice', age=5, height=80),
Row(name='Alice', age=5, height=80),
Row(name='Alice', age=10, height=80)]).toDF()
df.dropDuplicates().show()
df.dropDuplicates(['name', 'height']).show()