powershell - How can I call many URLs from a list asynchronously -

i have few hundred thousand urls need call. these calls application server process them , write status code table. not need wait response (success/fail), server got request. want able specify how many concurrent jobs can running @ once haven't worked out how many concurrent requests tomcat can handle.

here's i've got far, taken someone's else's attempt similar, not url calls. text file contains each url on own line. url looks this:

http://webserver:8080/app/mwo/services/create?server=servername&e1user=admin&newmwonum=123456&sourcemwonum=0&tagnum=33-a-1b 

and code:

$maxconcurrentjobs = 10 $content = get-content -path "c:\temp\urls.txt"  foreach ($url in $content) {     $running = @(get-job | where-object { $_.state -eq 'running' })     if ($running.count -le $maxconcurrentjobs) {         start-job {              invoke-webrequest -usebasicparsing -uri $using:url         }     } else {          $running | wait-job -any     }     get-job | receive-job } 

the problems i'm having is giving 2 errors per "job" , i'm not sure why. when dump url array $content looks fine , when run invoke-webrequest 1 one work without error.

126    job126          backgroundjob   running       true            localhost            ...                 invalid uri: hostname not parsed.     + categoryinfo          : notspecified: (:) [invoke-restmethod], uriformatexception     + fullyqualifiederrorid : system.uriformatexception,microsoft.powershell.commands.invokerestmethodcomman     d     + pscomputername        : localhost  invalid uri: hostname not parsed.     + categoryinfo          : notspecified: (:) [invoke-restmethod], uriformatexception     + fullyqualifiederrorid : system.uriformatexception,microsoft.powershell.commands.invokerestmethodcomman     d     + pscomputername        : localhost 

any or alternative implementations appreciated. i'm open not using powershell, i'm limited windows 7 desktops or windows 2008 r2 servers, , i'd running final script on server using localhost in url cut down on network delays.

with jobs incur large amount of overhead, because each new job spawns new process.

use runspaces instead!

$maxconcurrentjobs = 10 $content = get-content -path "c:\temp\urls.txt"  # create runspace pool $maxconcurrentjobs  # maximum number of runspaces allowed run concurrently     $runspace = [runspacefactory]::createrunspacepool(1,$maxconcurrentjobs)  # open runspace pool (very important) $runspace.open()  foreach ($url in $content) {     # create new powershell instance , tell execute in our runspace pool     $ps = [powershell]::create()     $ps.runspacepool = $runspace      # attach code     [void]$ps.addcommand("invoke-webrequest").addparameter("usebasicparsing",$true).addparameter("uri",$url)      # begin execution asynchronously (returns immediately)     [void]$ps.begininvoke()      # give feedback on how far     write-host ("initiated request {0}" -f $url) } 

as noted in linked serverfault post, can use more generic solution, invoke-parallel, above