how to drop the missing values in python code example
Example 1: Returns a new DataFrame omitting rows with null values
# Returns a new DataFrame omitting rows with null values
df4.na.drop().show()
# +---+------+-----+
# |age|height| name|
# +---+------+-----+
# | 10| 80|Alice|
# +---+------+-----+
Example 2: dropna threshold
#dropping columns having more than 50% missing values(1994/2==1000)
df=df.dropna(thresh=1000,axis=1)