New posts in Apache Spark

AWS Glue: How to add a column with the source filename in the output?

How to handle null values when writing to parquet from Spark

py4j.protocol.Py4JJavaError occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe

Overwrite only some partitions in a partitioned spark Dataset

overwrite hive partitions using spark

Converting epoch to datetime in PySpark data frame using udf

java.lang.IllegalArgumentException at org.apache.xbean.asm5.ClassReader.<init>(Unknown Source) with Java 10

How do I truncate a PySpark dataframe of timestamp type to the day?

Why is Kafka consumer ignoring my "earliest" directive in the auto.offset.reset parameter and thus not reading my topic from the absolute first event?

How to add days (as values of a column) to date?