python with spark programming model code example

Example 1: python with file.open

# Reference https://docs.python.org/3/library/functions.html#open

# Method 1
file = open("welcome.txt", "r") # mode can be r(read) w(write) and others 
data = file.read()
file.close()

# Method 2 - automatic close
with open("welcome.txt") as infile:
  data = file.read()

Example 2: entry point to programming Spark with the Dataset and DataFrame API

# entry point to programming Spark with the Dataset and DataFrame API

spark = SparkSession.builder.master(
  "local").appName(
  "Words Count").config(
  "spark.some.config.option", "some-value").getOrCreate()

Tags:

Python Example

Related