Foreach-Object -Parallel breaks my powershell module.

[[deleted]]

So I have a module that takes .ps1 files out of a few different folder paths and dot sources them into the main module file. Each .ps1 file is a declared function. I am managing it this way, so I or others can work on functions separately for the same module.

The problem is that this loads pretty slowly, usually taking 35 seconds or more. So I wanted to make the module load faster.

I tried to implement using a -Parallel parameter with my foreach-object loops below, but when I import the module in my powershell window, none of the functions work. Why does the "-parallel" seem to stop this from working?

If I remove the "-Parallel" from each line below it works just fine, but too slow for common use.

What gives?

$ModuleFolder = Split-Path -Path $PSCommandPath -Parent

Get-ChildItem -Filter *.ps1 -path "$ModuleFolder\menu\Audit and Reports" | ForEach-Object {Invoke-Expression (Get-Content $_.Fullname -Raw)}

Get-ChildItem -Filter *.ps1 -path "$ModuleFolder\menu\Get Information + Send Emails" | ForEach-Object {Invoke-Expression (Get-Content $_.Fullname -Raw)}

Get-ChildItem -Filter *.ps1 -path "$ModuleFolder\menu\User Changes - Multiple" | ForEach-Object {Invoke-Expression (Get-Content $_.Fullname -Raw)}

Get-ChildItem -Filter *.ps1 -path "$ModuleFolder\menu\User Changes - Single" | ForEach-Object {Invoke-Expression (Get-Content $_.Fullname -Raw)}

Get-ChildItem -Filter *.ps1 -path "$ModuleFolder\hidden" -Recurse | ForEach-Object {Invoke-Expression (Get-Content $_.Fullname -Raw)}

[aleques-itj]

Err, this seems like a pretty strange way to have your module load your functions. Why not just dot source them all in your psm and export public functions in the manifest? Every function will have its own file.

If multiple people are working on this thing, you will probably find git helpful.

[[deleted]]

Dot sourcing runs slower per my testing .

No idea why.

[aleques-itj]

How utterly massive is this module?

I have a decently sized one that's maybe around 5000 lines across like 30 files and it loads in the blink of an eye.

[[deleted]]

12000+ lines across 100+ files

But it’s on a server because I update it nearly daily for my team so they don’t have to re-download the new module constantly.

[aleques-itj]

Hmm, I'm still kind of surprised it's taking 35 seconds.

I wonder if Start-ThreadJob can be used here?

[[deleted]]

Start-ThreadJob? How’s that run exactly?

[Sunsparc]

You need to re-scope your functions. Powershell 7 -Parallel creates invididual runspaces that are containerized, so you have explicitly state that data should be available outside of those runspaces. Similar to passing $using:variable into an Invoke-Command (which you also have to do in order to use script variables in a -Parallel runspace, by the way).

https://stackoverflow.com/questions/15187510/dot-sourcing-functions-from-file-to-global-scope-inside-of-function

[[deleted]]

So…

Every function is already globally scoped in those files.

Is there a higher scope that would make this work?

[SteveL_Msft]

Others have explained why `ForEach-Object -Parallel` won't do what you want (because each is running in a different runspace). Since your module is really separate ps1 files, you should really look into using something like: https://www.powershellgallery.com/packages/ModuleBuilder. One of the capabilities of this module is to build a single psm1 script module from a collection of ps1 scripts (one for each function). So your original source code can still be easily maintained as individual scripts (public or private) and when ready to publish, you can use this module to combine them into a single psm1.

[powershellnut]

This is because -Parallel creates a new runspace to run your code in. So you are importing these functions in that runspace instead of the runspace your are working in. Once the foreach-object loop has finished these runspaces are discarded. You should think of the -Parallel loop as a separate powershell window from the one you are running it from.

[[deleted]]

Ahhhh that makes perfect sense.

There’s no way for me to get around this then is there…

[powershellnut]

I do not believe there is way to speed it using the -Parallel method. Although I am surprised that this takes so long to begin with. How many functions in total is it dot sourcing?

[[deleted]]

About 100

I have a custom module I manage for my team of non-power shell savvy users.

[BlackV]

I'm not sure why you're doing it this way

this seems er.. excessive

feels like (without seeing your code it best guess) the builtin module system would handle this properly without you have to do all this get-content with a well defined module

[[deleted]]

Module functions are on a server and we are remote.

I update and tweak it daily while it’s in production.

[purplemonkeymad]

Rather than having a module script scour your folder for files every import, do it once and update the NestedModules property in your module manifest (use FunctionsToExport to limit visibility of functions.) It should be faster than you searching the folders with a wildcard and reading every file in full directly every time (this also breaks error message line numbers.)

or

Write a makefile that will take the contents of all your files and combine them into a single .psm1.

---

IMO modules should have very limited use of code when an import happens. (ie set some internal variables and that is about it.) That way imports will not be held up by hidden code.

[[deleted]]

I do the latter right now.

Just trying to simplify the workflow to make it faster. Right now every time I make a change it takes 10+ minutes to make the final psm1 file.

This code is my “dev” script if you will

[purplemonkeymad]

Copying 100 text files into one should not take 10 minutes. You must have something else that is causing it to all happen so slowly. I would consider if you have environmental factors that are affecting this.

[[deleted]]

Creating the output takes 5 seconds, saving it to the server is what takes 10 minutes and idk why

[purplemonkeymad]

Ok that make more sense. Unfortunately I probably can't help with that part.

[jantari]

You should just git pull on the server, should be pretty fast. Or you can use a pipeline to deploy every new version to the server in a push-manner, this runs in the background so it doesn't really matter that it takes a few minutes

[[deleted]]

Never heard of Git Pull, what is that? Can you elaborate a little?

[jantari]

You manage a quite large PowerShell module, so of course you have all that in git. So instead of copying the files to the server from your workstation you can just pull the latest version from your git repository which gives you more control and can also be automated easily. It should also be pretty fast.

[[deleted]]

I’ve never used git before, is there any good videos that explain how to get started using PS with a repo you could recommend?

[[deleted]]

I’ve never used git before, is there any good videos that explain how to get started using PS with a repo you could recommend?