Difference between Spark RDD's take(1) and first()
Infact first is implemented in terms of take.
Following is taken from spark's source of RDD.scala. first calls take(1) and returns the first element if found.
def first(): T = withScope {
take(1) match {
case Array(t) => t
case _ => throw new UnsupportedOperationException("empty collection")
}
}
take(num) tries to take num elements from starting from RDD's 0th partition (if you consider 0 based indexes). So the behavior of take(1) and first will be identical.
Even the spark programming guide confirms this.
About your second question: it depends what you mean when you say partitioned differently. If you are calling sc.textFile("/path/to/file") with or without numPartitions, it wouldn't matter because 0th partition will always be 0th partition. So Yes, you can assume that they will have the same first element.
EDIT: Partitions in RDD are ordered, the physical first line in your CSV will end up in the 0th partition on RDD. And take(1) and first both will return that first row of 0th partition.
No both are not same.
rdd.first() will Return the first element in this RDD
while rdd.take(1) will return an array that will have first element only.
- Is the underlying implementation of first() the same as take(1)?
Ans : In terms of implementation first() calls take(1) internally and returns first and only element of the array returned by take(1). Taken from org.apache.spark.rdd.RDD class
/**
* Return the first element in this RDD.
*/
def first(): T = withScope {
take(1) match {
case Array(t) => t
case _ => throw new UnsupportedOperationException("empty collection")
}
}
- Suppose rdd1 and rdd2 are constructed from the same csv, can I safely assume that rdd1.take(1) and rdd2.first() will always return the same result, i.e., the first row of the csv? What if rdd1 and rdd2 are partitioned differently?
Ans : Yes you can assume, partitioning do not change the order in which input was read.