Simple way to monitor a multiple URLDownloadSubmit calls?
Here is an example of how to build a download manager.
We start by defining a function that uses URLDownloadSubmit
to initiate a download:
manifest = <||>;
SetAttributes[taskProgress, HoldFirst]
taskProgress[manifest_][event_] :=
manifest = <|manifest, event["Task"] -> event|>
SetAttributes[taskFinished, HoldFirst]
taskFinished[manifest_][event_] :=
manifest = <|manifest, event["Task"] -> event|>
SetAttributes[startJob, HoldFirst]
startJob[manifest_][src_, dest_] := URLDownloadSubmit[
src, dest,
HandlerFunctions -> <|
"ConnectionFailed" -> connectionFailed[manifest],
"CookiesReceived" -> cookiesReceived[manifest],
"HeadersReceived" -> headersReceived[manifest],
"TaskFinished" -> taskFinished[manifest],
"TaskProgress" -> taskProgress[manifest],
"TaskStatusChanged" -> taskStatusChanged[manifest]
|>,
HandlerFunctionsKeys -> {
"Task", "TaskStatus", "File",
"ByteCountTotal", "ByteCountDownloaded", "FractionComplete"
}
];
We have defined a variable, manifest
, that will hold information about the files being downloaded. It is up to the user to define the event handler functions that they want to use; in my download manager, I will only use TaskProgress
and TaskFinished
. Whenever any of those events are called, I update manifest
with the latest information. The latest information includes the variables specified under HandleFunctionsKeys
.
This is all we need, really. Now we can build an interface to visualize manifest
.
SetAttributes[abortDownload, HoldFirst]
abortDownload[manifest_, task_] := (
TaskRemove /@ Select[Tasks[], #["TaskUUID"] === task["TaskUUID"] &];
manifest = <|
manifest,
task -> <|manifest[task], "TaskStatus" -> "Aborted"|>
|>)
SetAttributes[visualizeManifest, HoldFirst]
visualizeManifest[manifest_] := TableForm[Join[
{{"File", "Size (MB)", "Downloaded (MB)", "Fraction complete",
"Status", ""}}, {
#File
, Floor[#ByteCountTotal/10^6]
, Floor[#ByteCountDownloaded/10^6]
, ProgressIndicator[#FractionComplete]
, #TaskStatus
, Button["Abort", abortDownload[manifest, #Task],
Enabled -> (#TaskStatus =!= "Aborted")]
} & /@ Values[manifest]
]]
I will also add a button to begin downloading an Anaconda installer. Anaconda is a software for Python programmers that I picked because the installer is large enough in size that the download won't finish in a blip.
i = 0;
Button["Download", startJob[manifest][
"https://repo.anaconda.com/archive/Anaconda3-5.2.0-MacOSX-x86_64.pkg",
"~/Downloads/anaconda" <> ToString[i++] <> ".pkg"
]]
Dynamic@visualizeManifest[manifest]
The final result looks like this:
You can easily compute other statistics such as how many of the files have finished downloading by going through the values in the manifest
association.
Here's another way to do this, based on extracting the "File"
handler key
parallelDownload[things_] :=
DynamicModule[
{jobs, results},
Dynamic[
Internal`LoadingPanel@
Grid@
If[AllTrue[Values@results, # =!= None &],
Append[
{
Button[
"Get Result",
NotebookWrite[
Nest[ParentBox, EvaluationBox[], 5],
ToBoxes@Values@results
]
],
SpanFromLeft
}
],
Identity
]@
KeyValueMap[
{
Row@{#[[1]], ":"},
If[#2 =!= None, "Complete", "Waiting..."]
} &,
results
]
],
Initialization :>
{
results = <||>,
jobs = <||>,
Map[
With[{job = Flatten[{#}]},
results[job] = None;
jobs[job] =
URLDownloadSubmit[
Sequence @@ job,
HandlerFunctions ->
<|"TaskFinished" :> Function[results[job] = #File]|>,
HandlerFunctionsKeys -> {"File"}
]
] &,
things
]
}
]
If we run that on the jobs it pops a little watching interface:
After that's done it shows a button to replace the box with the result:
And pressing the button gives us the desired result:
{
"~/Downloads/1.jpg", "~/Downloads/2.jpg",
"~/Downloads/3.jpg", "~/Downloads/4.jpg",
"~/Downloads/5.jpg"
}
With ProgressIndicator
Here's an alternate interface for many files
parallelDownloadDynamic[
things_,
var:Dynamic[_]|None:None,
e:OptionsPattern[Dynamic]
] :=
DynamicModule[
{jobs, results},
Dynamic[
Replace[var,
Verbatim[Dynamic][s_]:>(Set[s, results])
];
If[AllTrue[Values@results, # =!= None &],
Button[
"Get Result",
NotebookWrite[
Nest[ParentBox, EvaluationBox[], 1],
ToBoxes@Values@results
]
],
Internal`LoadingPanel@ProgressIndicator[
Count[Values[results], Except[None]],
{0, Length[results]}
]
],
e
],
Initialization :>
{
results = <||>,
jobs = <||>,
Map[
With[{job = Flatten[{#}]},
results[job] = None;
jobs[job] =
URLDownloadSubmit[
Sequence @@ job,
HandlerFunctions ->
<|"TaskFinished" :> Function[results[job] = Replace[#["File"], Except[_String]->$Failed]]|>,
HandlerFunctionsKeys -> {"File"}
]
] &,
things
]
}
]
You may use the HandlerFunctions
option of URLDownloadSubmit
to track a counter updated in "TaskFinished"
event.
With jobs
as defined in OP.
completedCount = 0;
tasks =
URLDownloadSubmit[#1, #2,
HandlerFunctions -> <|"TaskFinished" -> (completedCount++ &)|>
] & @@@ jobs;
ProgressIndicator[Dynamic[completedCount], {0, Length@tasks}]
This outputs a ProgressIndicator
that tracks the completion of the tasks.
Hope this helps.