R doParallel Progress bar to monitor finished jobs
I could not find a solution with doParallel (I don't think it supports progress bars for job completion), but maybe you can try the new pbabbly:
# pbapply solution
library(pbapply)
cl<-parallel::makeCluster(nthreads)
invisible(parallel::clusterExport(cl=cl,varlist=c("nreps")))
invisible(parallel::clusterEvalQ(cl=cl,library(utils)))
result<-pblapply(cl=cl,
X=1:nreps,
FUN=funrep)
parallel::stopCluster(cl)
(disclaim: I'm the author of the progressr package and the future framework)
The progressr package (currently only available on GitHub) can achieve this when using doFuture as a parallel backend to foreach:
library(progressr) ## use progressr for procession updates
library(doFuture) ## attaches also foreach and future
registerDoFuture() ## tell foreach to use futures
plan(multisession) ## parallelize over a local PSOCK cluster
xs <- 1:5
with_progress({
p <- progressor(along = xs) ## create a 5-step progressor
y <- foreach(x = xs) %dopar% {
p() ## signal a progression update
Sys.sleep(6.0-x)
sqrt(x)
}
})
The default is to use utils::txtProgressBar()
for progression reporting, but you can change this. For example, the following will make progression updates being reported via progress::progress_bar()
and beepr::beep()
:
progressr::handler("progress", "beepr")
You can also add messages for each progression update, e.g.
p(sprintf("x=%g", x))
FYI, plan(multisession, workers = 2)
is short for plan(cluster, workers = cl)
where cl
is basically cl <- parallel::makeCluster(2L)
.
PS. The objective of the progressr package is to provide a minimal, sustainable, extendable, and unified API for progression updates. This, while being invariant to what iterator framework is used.
PPS. The progressr API is under development; it might take a while before it has identified its true self.