How to add custom description to Spark Job for displaying in Spark Web UI
use the sc.setJobGroup
:
Examples:
python:
In [28]: sc.setJobGroup("my job group id", "job description goes here")
In [29]: lines = sc.parallelize([1,2,3,4])
In [30]: lines.count()
Out[30]: 4
Scala:
scala> sc.setJobGroup("my job group id", "job description goes here")
scala> val lines = sc.parallelize(List(1,2,3,4))
scala> lines.count()
res3: Long = 4
SparkUI:
I hope this is what you are looking for.
Note that new Zeppelin 0.8 loose his tracking hook if you change the JobGroup name, and can't display his job progress bar (job still working, no effect on the job itself)
You can use
sc.setLocalProperty("callSite.short","my job description")
sc.setLocalProperty("callSite.long","my job details long description")
instead
See How to change job/stage description in web UI? for some screen captures and scala syntax