create new dataframe spark code example
Example 1: create spark dataframe in python
spark.createDataFrame(["10","11","13"], "string").toDF("age")
Example 2: create spark dataframe from pandas
import numpy as np
import pandas as pd
# Enable Arrow-based columnar data transfers
spark.conf.set("spark.sql.execution.arrow.enabled", "true")
# Generate a pandas DataFrame
pdf = pd.DataFrame(np.random.rand(100, 3))
# Create a Spark DataFrame from a pandas DataFrame using Arrow
df = spark.createDataFrame(pdf)