Could not bind on a random free port error while trying to connect to spark master
Set spark.driver.bindAddress
to your local IP like 127.0.0.1
.
pyspark -c spark.driver.bindAddress=127.0.0.1
While creating spark session set the below configurations
spark =SparkSession.builder.appName(str(name))\
.master("local[*]").config("spark.driver.memory","5g")\
.config("spark.driver.host","10.51.4.110")\ #Machine ip
.config("spark.driver.bindAddress","10.51.4.110")\ #Machine ip
.getOrCreate()
Sometimes In addition to bind address, we need to set the host address also. In my case system host address changed to system name. Spark shows timeout error. then after setting host and bind address as same. It works fine
*10.51.4.110 - Local Machine IP