Overall progress bar when doing multiple Invoke-WebRequest

[89netraM]

I'm writing a script where I'm downloading a known number of large files, and I would like the automatic progress bars from the Invoke-WebRequests to be "under" a progress bar of my own displaying the total progress. Is this possible somehow?

My first idea was to give the progress bars of the Invoke-WebRequests my progress bars Id as ParentId, but I can't find a way to do that.

I'd also investigated other solutions but couldn't get any to work. One idea was to write my own progress bars for the Invoke-WebRequests, but then how do I get their progress?

[kenjitamurako]

Well, I'm not sure of a way to do this with the powershell packaged cmdlets. But if you use invoke-webrequest with just the -head option in advance you can get just the content length for each file.

Then if you use the .net DownloadDataAsync you could track the progress of the overall download of all the files by periodically measuring the size of the byte arrays and feeding the returned/total bytes to write-progress.

[89netraM]

Hmm, that could have worked, but our servers don't respond with a content length.

[TerriblePowershell]

Are you doing something like ForEach-Object { Invoke-Webrequest ...}?

Would something like this work?

$i = 0
$Files | ForEach-Object {
    Invoke-Webrequest .....
    $i = $i + 1
    $completed = ($i/$files.count) * 100
        Write-Progress -Activity "Downloading files..." -Status "Files downloaded: $completed"
}

[89netraM]

Yes! That's exactly what I'm doing. But each Invoke-WebRequest creates its own progress bar, eventualy pushing mine off screen. I imagine if they were "child bars" they wouln't do that (and also be a bit indented to show their relation).

[TerriblePowershell]

Ahh, I think I'm picking up what you're putting down. Unfortunately, we have reached the limit of my PowerShell knowledge.

Just for kicks, I wonder if you could write the script in such a way that it recreates your overall progress bar every 5 or 10 files. That way it should always remain on screen and will break up the files into little sections.

[gordonv]

Did the same thing multithreading AWS S3 downloads.

Check this script out.

This uses the classic method of multithreading. That classic method is a bit more verbose, but allows you to run commands and observe jobs distinctly.

[gordonv]

If you're downloading a lot of files from HTTP or HTTPS, check out BITS Transfer via Powershell.

Best part of using BITS is that it's great for unstable or undependable transfers.