pyspark sql query for age field in a dataframe age < 40 code example

Example 1: Returns a DataFrame representing the result of the given query

# Returns a DataFrame representing the result of the given query

df.createOrReplaceTempView("table1")
df2 = spark.sql("SELECT field1 AS f1, field2 as f2 from table1")
df2.collect()
# [Row(f1=1, f2='row1'), Row(f1=2, f2='row2'), Row(f1=3, f2='row3')]

Example 2: Create a DataFrame with single pyspark.sql.types.LongType column named id, containing elements in a range

# Create a DataFrame with single column named id, containing elements in a range

spark.range(1, 7, 2).collect()
# [Row(id=1), Row(id=3), Row(id=5)]

spark.range(3).collect()
# [Row(id=0), Row(id=1), Row(id=2)]