Aggregate the elements of each partition, and then the results for all the partitions code example
Example: Aggregate the elements of each partition, and then the results for all the partitions
#Aggregate the elements of each partition, and then the results for all the partitions
seqOp = (lambda x, y: (x[0] + y, x[1] + 1))
combOp = (lambda x, y: (x[0] + y[0], x[1], y[1]))
sc.parallelize([1, 2, 3, 4]).aggregate((0, 0), seqOp, combOp)
# (10, 4)
sc.parallelize([]).aggregate((0, 0), seqOp, combOp)
# (0, 0)