pyspark: 'PipelinedRDD' object is not iterable
RDD can iterated by using map and lambda functions. I have iterated through Pipelined RDD using the below method
lines1 = sc.textFile("\..\file1.csv")
lines2 = sc.textFile("\..\file2.csv")
pairs1 = lines1.map(lambda s: (int(s), 'file1'))
pairs2 = lines2.map(lambda s: (int(s), 'file2'))
pair_result = pairs1.union(pairs2)
pair_result.reduceByKey(lambda a, b: a + ','+ b)
result = pair.map(lambda l: tuple(l[:1]) + tuple(l[1].split(',')))
result_ll = [list(elem) for elem in result]
===> result_ll = [list(elem) for elem in result]
TypeError: 'PipelinedRDD' object is not iterable
Instead of this I replaced the iteration using map function
result_ll = result.map( lambda elem: list(elem))
Hope this helps to modify your code accordingly