How can I call many URLs from a list asynchronously
With Jobs you incur a large amount of overhead, because each new Job spawns a new process.
Use Runspaces instead!
$maxConcurrentJobs = 10
$content = Get-Content -Path "C:\Temp\urls.txt"
# Create a runspace pool where $maxConcurrentJobs is the
# maximum number of runspaces allowed to run concurrently
$Runspace = [runspacefactory]::CreateRunspacePool(1,$maxConcurrentJobs)
# Open the runspace pool (very important)
$Runspace.Open()
foreach ($url in $content) {
# Create a new PowerShell instance and tell it to execute in our runspace pool
$ps = [powershell]::Create()
$ps.RunspacePool = $Runspace
# Attach some code to it
[void]$ps.AddCommand("Invoke-WebRequest").AddParameter("UseBasicParsing",$true).AddParameter("Uri",$url)
# Begin execution asynchronously (returns immediately)
[void]$ps.BeginInvoke()
# Give feedback on how far we are
Write-Host ("Initiated request for {0}" -f $url)
}
As noted in the linked ServerFault post, you can also use a more generic solution, like Invoke-Parallel
, which basically does the above