How I know the runtime of a code in scala?
Starting from Spark2+
we can use spark.time(<command>)
(only in scala until now) to get the time taken to execute the action/transformation..
Example:
Finding count of records in a dataframe
scala> spark.time(
sc.parallelize(Seq("foo","bar")).toDF().count() //create df and count
)
Time taken: 54 ms //total time for the execution
res76: Long = 2 //count of records
Based on discussion here, you'll want to use System.nanoTime
to measure the elapsed time difference:
val t1 = System.nanoTime
/* your code */
val duration = (System.nanoTime - t1) / 1e9d