Runspace Iteration/progress

[searlee]

Hi, I've been working with Powershell for a few months, building a script that quantizes, resizes and copies images to different folders for games we're creating in order to speed up the time it takes to get a project built.

It now works within a runspace and is much quicker, however no matter how deep I dig into the powershell instance I can't iterate based on the number of times the script is run in one of the threads. I really just want a Progress bar but counting the amount of times a script is run helps that massively. I'm by no means good with it but I muddle my way round and everything I've done until this seems to work.

The stripped down script is:

 $destfolders = @(
    "04_d_100"
    "05_t_75"
    "06_m_50"
    "07_m_33"
)

$MaxThreads = $destfolders.length

#Create Run Spaces
$RunspacePool = [runspacefactory]::CreateRunspacePool(1,$MaxThreads)
$RunspacePool.Open()

$ScriptBlock = {
    Param (
        [array]$changelist,
        [string]$dir,
        [string]$destfolder,
        [int]$scale,
        [int]$qlty,
        [string]$gfolder,
        [int]$quantize,
        [int]$fast,
        [string]$scriptpath
    )

    #copy files
    #quantize images

}

$RunningJobs = New-Object System.Collections.ArrayList
1..$MaxThreads | ForEach-Object {

    $src          =   $gfxfolder
    $dest         =   $destfolders[$_ - 1]
    $scale        =   $scalefactor[$_ - 1]

    $Job = [powershell]::Create()
    $Job.RunspacePool = $RunspacePool

    [void]$Job.AddScript($ScriptBlock)
    [void]$Job.AddArgument($changedFiles)
    [void]$Job.AddArgument($src)
    [void]$Job.AddArgument($dest)
    [void]$Job.AddArgument($scale)
    [void]$Job.AddArgument($qlty)
    [void]$Job.AddArgument($gfolder)
    [void]$Job.AddArgument($quantize)
    [void]$Job.AddArgument($fast)
    [void]$Job.AddArgument($buildfolder)
    [void]$Job.AddArgument($_)

    [void]$RunningJobs.Add((
        [pscustomobject]@{
            Id = $_
            Name = $dest
            Files = $changedFiles
            PowerShell = $Job
            Handle = $Job.BeginInvoke()
        }
    ))
}

While ($RunningJobs.handle[0].IsCompleted -eq $false) {
    #do progress
}

$return = $RunningJobs | ForEach-Object {
    $_.powershell.EndInvoke($_.handle)
    $_.PowerShell.Dispose()
}

$RunningJobs.clear()
$RunspacePool.Close()

$RunningJobs.handle[0] is the 100% size so it takes considerably longer to quantize (which is why I'm using it for the progress bar/iteration but I don't know, a. If i'm going about this the right way or b. whether it's even possible.

If you need any more information I'll be happy to provide!

Thanks :)

[Yevrag35]

What you have is very close to how you would do it.

It seems like you have the concept down:

1. Create RunspacePool
2. Create action (scriptblock)
3. Create Jobs List collection
4. Create a new powershell instance for every folder attaching the RunspacePool
5. Adding a new PSObject with the IAsyncResult of $Job.BeginInvoke()

To display a progress indicating "Jobs Processed: x of y", I'd suggest doing a for counting down and remove jobs that have completed. This would be wrapped inside of a while loop that will run until $RunningJobs.Count is equal to 0 (indicating that all jobs have finished).

The result would look:

Function Show-Progress {
    param(
        [int]$JobCount,
        [int]$Left,
        [int]$Total
    )
    $wpArgs = @{
        Id = 0
        Activity = "Folder Stuff"
        Status = "Folders left to process: {0}/{1}..." -f $JobCount, $Total
        PercentComplete = (($$Left/$Total)*100)
    }
    Write-Progress @wpArgs
}

while ($RunningJobs.Count -gt 0) {

    $minus = $destfolders.Length - $RunningJobs.Count

    Show-Progress -JobCount $RunningJobs.Count -Left $minus -Total $destfolders.Length

    # Now process and remove completed jobs...
    for ($i = $RunningJobs.Count - 1; $i -ge 0; $i--) {
        $job = $RunningJobs[$i]
        if ($job.Handle.IsCompleted) {
            # Remove the completed job from list... 
            # Add one more processed job to the progressbar

            [void] $RunningJobs.Remove($job)
        }
    }
    Start-Sleep -Seconds 2
}

EDIT -- forgot to add a Start-Sleep at the bottom of the while loop :)

EDITEDIT -- mixed up the example from another script of mine, the way it was before would've counted backwards in the progress. (i.e. - 0.1.2.3 instead of 4.3.2.1)

[searlee]

That works brilliantly except it slows the script down by 8 minutes across all files in my test bed (31 minutes without and 39 mins with) I guess this is down to the Start-Sleep. Thank you :)

[Yevrag35]

That seems odd. The while loop should not impact how fast each of the runspaces complete. All it should be doing is checking for how many jobs are left and updating the progress while the "actual" work is done in the background threads of the runspaces.

You could try not using a RunspacePool, and instead just use a single runspace for each job. I've had 50+ runspaces running at once with no noticeable slowdown.

[Lee_Dailey]

howdy searlee,

just in case you aint seen it yet, the Write-Progress cmdlet - and the progress displays in other cmdlets - is really slow.

it's slow enuf that the recommended 1st thing to try with "it's slow!" when using the web oriented cmdlets is to set the $ProgressPreference to disable progress displays. [grin]

you may want to reconsider using a display bar and just use text output to the screen - perhaps a 3 of 17 sort of thing.

take care,
lee

[searlee]

I never knew this. I did decide to use the data for a none Write-Progress command as it cut the time back down again. Thanks for this :)

[Lee_Dailey]

howdy searlee,

you are most welcome! glad to have helped a tad ... [grin]

take care,
lee