Automatically including jars to PySpark classpath

You can add the jar files in the spark-defaults.conf file (located in the conf folder of your spark installation). If there is more than one entry in the jars list, use : as separator.

spark.driver.extraClassPath /path/to/my.jar

This property is documented in https://spark.apache.org/docs/1.3.1/configuration.html#runtime-environment

As far as I know, you have to import jars to both driver AND executor. So, you need to edit conf/spark-defaults.conf adding both lines below.

spark.driver.extraClassPath /path/to/my.jar
spark.executor.extraClassPath /path/to/my.jar

When I went through this, I did not need any other parameters. I guess you will not need them too.