pandas dataframe remove duplicate rows code example

Example 1: remove duplicate row in df

df = df.drop_duplicates()

Example 2: df remove duplicate rows

df = df.drop_duplicates()
p

Example 3: python remove duplicates

word = input().split()

for i in word:
  if word.count(i) > 1:
    word.remove(i)

Example 4: remove duplicate values in data frame r

df <- df %>% distinct()

Example 5: Return a new DataFrame with duplicate rows removed

# Return a new DataFrame with duplicate rows removed

from pyspark.sql import Row
df = sc.parallelize([
  Row(name='Alice', age=5, height=80),
  Row(name='Alice', age=5, height=80),
  Row(name='Alice', age=10, height=80)]).toDF()
df.dropDuplicates().show()
# +---+------+-----+
# |age|height| name|
# +---+------+-----+
# |  5|    80|Alice|
# | 10|    80|Alice|
# +---+------+-----+

df.dropDuplicates(['name', 'height']).show()
# +---+------+-----+
# |age|height| name|
# +---+------+-----+
# |  5|    80|Alice|
# +---+------+-----+