Spark: what options can be passed with DataFrame.saveAsTable or DataFrameWriter.options?
The reason you don't see options
documented anywhere is that they are format-specific and developers can keep creating custom write formats with a new set of options
.
However, for few supported formats I have listed the options as mentioned in the spark code itself:
- CSVOptions
- JDBCOptions
- JSONOptions
- ParquetOptions
- TextOptions
- OrcOptions
- AvroOptions
Take a look at https://github.com/delta-io/delta/blob/master/src/main/scala/org/apache/spark/sql/delta/DeltaOptions.scala the class "DeltaOptions'
Currently, supported options are:
- replaceWhere
- mergeSchema
- overwriteSchema
- maxFilesPerTrigger
- excludeRegex
- ignoreFileDeletion
- ignoreChanges
- ignoreDeletes
- optimizeWrite
- dataChange
- queryName
- checkpointLocation
- path
- timestampAsOf
- versionAsOf