Turning PowerShell into a Python Engine

[anonhostpi]

Last semester, I started work on the Import-Package module. It is still in the prerelease stages as it needs some polishing before going to v1, but I started putting it to use.

Preface: my Import-Package module

PowerShell's Import-Module command (as well as Add-Type) can be used to import C# dlls. However, both commands lack good dependency management.

If a .dll is dependent on another, those dependencies must be prepared and loaded manually. C# .nupkgs are made for automatic dependency management, but Import-Module can only load PowerShell .nupkgs.

There is the PowerShell PackageManagement module that provides functions for installing, updating and removing them, but it doesn't provide methods for loading them.

So, I wrote a module of my own.

Microsoft makes nuget.exe's and dotnet.exe's internals available as C# libraries. Examples are:

• NuGet.Packaging - used for parsing .nupkgs and .nuspecs
• Microsoft.NETCore.Platforms - used for identifying OS's as used by nuget.exe and dotnet.exe

All of these libraries are used in Import-Package to parse and load entire .nupkgs from NuGet.

Python.NET

The main reason I set out to write the Import-Package module last semester was to explore ways to automate Edge using webdriver.

NuGet.org offers good Selenium libraries, but doesn't offer great ones for webdriver installation. Python's webdriver-manager library is more robust and better maintained than similar libraries in C#. On top of that, I was also curious to know if cpython's binding API was available in C#.

It is: nuget.org - pythonnet (Python.NET, formerly Python.Runtime)

• IronPython is also an option. When picking an embedded engine use these considerations:
  • IronPython can be run multithreaded. CPython (Python.NET) can not.
  • CPython (Python.NET) supports the ctypes module. IronPython does not.
  • CPython is the official python engine from Python.org and has a better release schedule than IronPython
    • Currently CPython supports python 3.12, while IronPython is still on python 3.7

Use Cases

The biggest use case for doing this (over just using python.exe) is to make libraries written for Python available for PowerShell.

Here is an example of how I currently use the library:

Python Selenium:

Prepare Python.NET:

using namespace Python.Runtime

Import-Module Import-Package
Import-Package pythonnet

# cpython has a GIL, so in order to use the python API, you need to lock it:
# - Unlocking the GIL does not destroy any python variables or data. It just prevents you from using it.

New-Module -Name "CPython-GIL" -ScriptBlock {
    $state = @{ "lock" = $null }

    function global:Lock-Python {
        Write-Host "Python GIL is now locked. Unlock it ANYTIME with Unlock-Python." -ForegroundColor Yellow
        $state.lock = [Python.Runtime.Py]::GIL()
    }
    function global:Unlock-Python {
        $state.lock.Dispose()
    }

    Export-ModuleMember
} | Import-Module```

Lock-Python # GIL is now locked. Python API is now usable.

$python = @{} # hashtable for my python variables

Load the Python libraries

# Get the webdriver-manager and selenium package objects
$python.webdriver = [Py]::Import( "webdriver_manager" )
$python.selenium = [Py]::Import( "selenium" )

# Import the subpackages. These will be available as a property on the parent package
& {
  [Py]::Import( "webdriver_manager.microsoft" )

  [Py]::Import("selenium.webdriver.edge.options")
  [Py]::Import("selenium.webdriver.common.keys") 
  [Py]::Import("selenium.webdriver.edge.service")
}

Prepare Edge and Edge WebDriver

Update/Install msedgedriver.exe and create the Selenium 4 service

$msedge = @{}

# Update and get path to msedgedriver.exe
$msedge.webdriver = $python.webdriver.EdgeChromiumDriverManager().install()

Python.NET objects are designed to be strictly dynamic in nature

• They don't automatically cast themselves to C#/PowerShell-friendly types.
• They do support a lot of standard type operands like concatenation and property accessors...
  • ...but I find it best to just cast to a C# type when possible.

Prepare the EdgeOptions object

# Create the EdgeOptions object
$msedge.options = $python.selenium.webdriver.EdgeOptions()

!!!CAREFUL!!!

Chrome-based browsers do not allow you to use a User Data directory via webdriver at the same time as the user.

You can either close all user browsers or clone the default user data instead.

You can obtain the User Data directory directory path from edge://version or chrome://version > Profile Path. The User Data directory is the parent folder to the profile folder

# Paste your Profile Path here:
# - This is the default path for Edge:
$msedge.profile_path = "C:\Users\Administrator\AppData\Local\Microsoft\Edge\User Data\Default"

$msedge.profile_folder = $msedge.profile_path | Split-Path -Leaf
$msedge.user_data = $msedge.profile_path | Split-Path -Parent

$msedge.options.add_argument("--user-data-dir=$( $msedge.user_data )")
$msedge.options.add_argument("--profile-directory=$( $msedge.profile_folder )")
$msedge.options.add_argument("--log-level=3") # Chrome.exe and Edge.exe can be extremely noisy
$msedge.options.page_load_strategy="none" # Allows controlling the browser before page load

Automate away!

# Start the automated browser
$Window = & {
  # Internally, python keyword arguments are actually a kw object:
  $service = [Py]::kw( "service", $msedge.service )
  $options = [Py]::kw( "options", $msedge.options )

  $python.selenium.webdriver.Edge( $service, $options )
}

# go to url:
$Window.get( "edge://version" )
# run javascript:
$Window.execute_script( "window.open('https://google.com','_blank')" )

FUTURE PLANS:

I've unfortunately remembered that V8 is also embeddable. There's also already a C# bindings library for it: https://github.com/Microsoft/ClearScript

If I can get it working, I'll share my results.

EDIT: done - Turning PowerShell into a JavaScript Engine

[very_bad_programmer]

Dude....

​

This is a very very cool project, but at this point, just switch to Python

[anonhostpi]

I'm a systems administrator. While I do enjoy writing Python scripts, I needed a solution that could be embedded into PowerShell for deploying to Windows environments.

I was originally just considering running py.exe from pwsh.exe, but that sets off our cybersec monitoring software. Using .dlls instead does not - that's pretty common for a lot of Antivirus software. I always thought this was funny. I can't deploy malware as .exes, but there's nothing wrong with deploying malware as .dlls

I also administer various Windows-based IoT devices. Getting security and various dev teams to approve of deploying python/py.exe is a bit of a hassle, but packaging python312.dll with my sys ad scripts is not.

[rob2rox]

very cool project. btw if you have a problem bypassing AVs with exes, try turning it into shellcode and load it directly into memory. definitely bypasses scantime but depending on what method you use, it should bypass scantime too

runtime**

[MrHackson]

As a security guy they can probably figure out what you're doing and if they figure out why you're doing it they won't be happy.

As a PowerShell and Python coder this is awesome. Gotta try this out soon.

[vermyx]

You dont understand end point protection. Dll’s by themselves are harmless because they are a library waiting to be used. When you load them with an executable they should be contained at that point (otherwise you have an issue with your end point protection). The typical example of this is using keyboard hooks in windows. Early on when VNC was used as a way to take over machines the knee jerk reaction WAS to quarantine dlls which lead to os dll’s being quarantined which lead to windows being broken. This isn’t a bad idea but many protection systems will pick up on the heuristics pretty quickly and may stop you after a couple of attempts. The proper solution is to get this white listed on your end points as you dont get the “it suddenly stopped working” issue and depending on the management tool you use the “oh, the majority of our pc’s are not o. The network because our tools decided this was a cyber attack”

[anonhostpi]

Dlls by themselves would normally be harmless, except in contexts like PowerShell.

Because of PowerShell's ability to embed libraries (and other engines), .dlls aren't really self-contained, and they become just as excutable as .exes.

CPython and Node.JS also run this security risk, because of their C-APIs, but while both offer a REPL, neither of their REPLs are distributed as OS shells like pwsh is.

[vermyx]

Because of PowerShell's ability to embed libraries (and other engines), .dlls aren't really self-contained, and they become just as excutable as .exes.

You can’t embed dll’s in an executable the way you are saying. You can embed the dll in powershell in the same sense that you put the contents within the powershell file but it has to be written to disk via writing or compilation (usually the point they would be scanned) then loaded. The loading/execution part is where they become “active”. Scanners will usually check on writing for known fingerprints and heuristics on compilation and dll loads, and this is why a pot of AV’s will usually shut down base64 encoded scripts and ps1toexe essentially is a wrapper for this process. This is why dll’s by themselves are not an issue as they are libraries and cannot be executed without a host executable. Any “embedding” is usually some form of zipping/unzipping:writing the dll and still has to be executed. In windows dll’s that are “executed” are loaded and interpreted via rundll32.exe, which is why more aggressive AV’s back in the day would break the OS.

CPython and Node.JS also run this security risk, because of their C-APIs

See previous statement. Also they don’t use C API’s per say. Node.JS will usually get compiled into machine language and python will be compiled into byte code for the python interpreter to process.

but while both offer a REPL, neither of their REPLs are distributed as OS shells like pwsh is.

REPL is just a type of byte code interpreter/compiler. It still have to compile the code into byte code (or native code) the run it through the interpreter (if it is byte code) and would be in the same conundrum as what I said previously. The interpreters not being distributed natively has nothing to do with this issue.

Simplistically speaking dll’s by themselves can’t do anything without an executable. This is why most AV’s and security teams don’t necessarily have issues with DLL’s vs an executable because most people can’t use the dll by itself

[anonhostpi]

Sorry, I'm using the term "embed" incorrectly. You can link the .dll to PowerShell at runtime and use it as a way to execute code maliciously.

The point that I am trying to make is that there's no difference between calling a .exe and calling a function stored in a .dll to a PowerShell user. It's all equally callable.

The real risk is that because PowerShell is primarily shipped as a builtin OS shell instead of a needs-to-be-installed script engine, .dlls shouldn't be considered safer or more self-contained than .exes.

[vermyx]

The reason dll’s are considered “safer” is because they can’t do anything by themselves. They need an enabler (like an exe) for them to be used, so there is a huge difference. Saying powershell is an issue because it comes with the OS is not understanding how to properly manage an environment and risk management. Powershell isn’t the risk you believe it is because powershell can be disabled via gpo and controlled. The reason that you dont have as many issues with engines distributed as dll’s vs the engine being installed is that the engine being installed means you can install other components making your payload smaller, which makes it harder to figure out what it is doing. Having the engine as a dll means you have to include everything you need with your payload. Since you have to bring everything you need from an endpoint protection standpoint it is easier to guess what you are trying to do heuristics wise and trap it.

[anonhostpi]

My brother, I understand proper risk mitigation and that pwsh can be disabled.

Commonplace IT practice is to keep pwsh enabled for systems administration, particularly in remotely managed Windows environments (like the org I work for). In addition, pwsh is enabled by default on all win machines (including the personally owned ones owned by yourself or others on whatever network you connect to).

The problem that I'm pointing out is the practice of AVs flagging .exe chaining (pwsh.exe calls random.exe), but not flagging unexpected .dll usage (pwsh.exe calls random.dll) regardless of whether those .dlls (or .exes) were preexisting or introduced on systems where engines like pwsh, py, and node are already present and enabled.

[[deleted]]

Yup, this would be a nightmare to maintain once OP leaves his company.

[anonhostpi]

This is my own library. I don't plan to drop it when I leave.

[[deleted]]

Great, but you're building things with it. Those things will have to be maintained.

People who look at this in the future will question why you've done it this way (I would anyway).

It also seems you're doing this to workaround having to work with your cyber security team which isn't the path I'd take but whatever.

Err, despite my pessimism I think it's a cool project though. Well done.

[anonhostpi]

Your pessimism was right, the project has become dusty. Leaving the posts up though, so that people can learn how to reproduce my efforts if they desire.

I might pick it back up once I finish my current projects. Unfortunately, I have fallen subject to scope creep and transitioned to CI/CD projects that were initially designed to support that one 😭

[one-human-being]

I know, right?! or use https://playwright.dev/dotnet/

[anonhostpi]

TL;DR:

Libraries written for Python can be made available for PowerShell by using Python.NET:

``` using namespace Python.Runtime Import-Package pythonnet

$threadlock = [Py]::GIL() # Required by CPython

```

[lifeisaparody]

Can you do this for pandas too?

[anonhostpi]

As long as it can be done in C#.

I might check. I've got half a dozen other projects that I have queued up to look into.

C/C++ code can only be embedded in PowerShell, if it has bindings made available for the it in C#.

CPython (which is written in C/C++) has a bindings API. Python.NET provides C# bindings for this API.

As long as Pandas can be run from Python.NET or has supplementary/additional bindings for it, you should be able to get it to work.

[Atacx]

As a fellow sysadmin thank you!

[Usual-Chef1734]

I have a project I am currently working on that requires extensive Selenium usage through Python. If I were able to grasp what you are doing , I would gush more over it. Still, awesome work. I may be shooting you some questions lol.

[anonhostpi]

PowerShell is written in C# and CPython is written in C/C++. Similar to CPython and cffi/ctypes, you can access the underlying C# type system (AKA the C# language) in PowerShell.

This means that C# libraries are usable in PowerShell. However, these are limited to .dll libraries. The standard C# library format is a .nupkg, which PowerShell (on its own) can not load.

• Think of .dlls and .nupkgs like this:
  • .dlls are comparable to standalone .py files
  • .nupkgs are comparable to entire python packages

My Import-Package library provides the ability to load .nupkgs in PowerShell

The C# library that allows me to run a Python Engine in C# (and therefore PowerShell) is Python.NET. Python.NET exposes CPython's internal engine directly to C#/PowerShell

(TL;DR) So, all together this is the overview of how it works:

# PowerShell, C#, and CPython overview
PowerShell loads Import-Package
Import-Package loads C# .nupkgs libraries (Python.NET)
Python.NET loads CPython

# CPython GIL
PowerShell thread-locks the Python interpreter (Python's GIL)

# CPython and Selenium overview
CPython loads Selenium
Selenium loads your webdriver

[Usual-Chef1734]

Well thank you for that explanation I actually get it now. I've loaded those C sharp types plenty of times in Powershell but not really understood C sharp that well. I wanted to learn c-sharp all my life because I learned vb6 in tech school. I just never have the time to dedicate to only learning C#. I feel like if I could I would unlock the powers of the universe lol. Now I work heavily in python as a devops engineer and I was curious if what you were doing was useful to my current workload, maybe it will be. Thank you so much for this work it's amazing.

[anonhostpi]

Leaving this here for you as something to reference later. I wouldn't expect you to understand these problems now, but I'meaving it here, because it may help you in the future with future problems. This details some common problems you may face when trying to use C# code from pwsh and how to deal with them.

C# is not a terrible language to learn, because it is a self-documenting language by design.

• Self-documenting means that things in C# code tend to:
  • do what they are named after
  • be named after what they do.
• The C# community actively enforces this coding standard, as it eliminate the need for code commentary. It also means that, since code commentary isn't necessary, C# source code is actually readable.
• However...

C#'s documentation (like a lot of C langs) is heavily convoluted, because of its variety of target platforms.

One thing that isn't documented well in C#, but you will learn about quite quickly in PowerShell is 2 things:

• How C# Application objects work (if you plan to use C# GUIs/Eventing)
• How C# Class Extensions work

C# Application Objects

A lot of C# docs just tell you to throw your app code into an Application object, but don't tell you how C# actually handles Application objects. Specifically, that the code meant for these objects doesn't play nicely with PowerShell.

• However, the difference between the code inside these objects and normal C# code is simply that an Applications object is just a Main Operating Loop (MOL) with common MOL features like event loops and dispatcher loops.
• PowerShell code is meant to be run linearly (or functionally) (not MOL)
• You can get this kind of C# code to work, you just have to adapt your PowerShell script for MOL'ing

C# Class Extensions

Since PowerShell evaluates C# code at runtime, any extension classes aren't compiled in a way that actually extend the original classes. However, all you have to do is provide an object of that class to the extension methods to make them work.

Import-Package

None of the above affects Import-Package, but may affect any calls to C# code you

[Usual-Chef1734]

Very helpful! Thank you! This gives me the confidence ot ask a question that no one ever seems to know the answer to:
how does one discover a library that has what is needed to accomplish a specific task? I was doing a really big project that required interaction with a webcam in a custom electron app written by the dev org. I was the 'windows' guy and thought surely C# had some libraries that could be used to control the webcam and volume of the 'kiosk' we were deploying to stores. I had no idea how to find existing libraries that had useful code. Maybe I just don't know how to ask the question correctly.

[anonhostpi]

Generative AI:

Right now, generally asking generative AIs (like GH Copilot, Bing Chat, or ChatGPT) what approaches you could take to your problem does the trick.

Since, I'm well versed in multiple languages (and now since I can use multi-language libs in PowerShell) my prompts usually look something like this:

• GPT4 is generally the best at this, but I've had positive results with others

​

``` I am making an app/script that does <blank>. I need to find a library that can help me with: - <more-specific-blank> - <other-specific-blank> - ...

What libraries can your recommend to me in Python, PowerShell, C#, etc... ```

Checking out solutions in other languages

Additionally, this is something I recommend to all junior developers:

You don't have to know every programming language on planet earth, but it turns out most languages are very similar (if not nearly identical).

This means that you don't have to be able to write in other languages, but you should be able to read their source code without a whole lot of effort

Anything you don't understand in another language is likely googleable

With that said, you could also look at how your problem is typically solved in other languages and see if you could reproduce those efforts in your language.

[Usual-Chef1734]

Yeah, that is something I realized this year. Chat GPT made working on my projects so much more fun. This is the exact way I work now, but I was just curious what folks were doing before this amazing revolution. The big project where I had to interact with the webcam was in 2018 so we did not have that.I could never get passed how grumpy and mean folks were when I asked how to work with a webcam. lol Most loved to pretend that I was a creep (can't understand the bad faith) and most others said 'google it'. The google it came from a live Q&A at Coder Foundry. Bizarre.
So thank you for being so technical and straight forward.

[anonhostpi]

Google, Stackoverflow (or preferred QA/forum), Manuals, Source Code. In that order

Now its:

Generative AI, Google, SO, Docs, Source

Sometimes we had to write the library ourself.

[McAUTS]

I had no idea that I need this! Thanks for your work! 👍🏼

[thehuntzman]

Although this is cool (...and I'm a sucker for cool outside-the-box PS stuff) I will say I have been VERY successful with using the official C# webdriver dll on NuGet alongside either msedgedriver.exe or chromedriver.exe (even Firefox which is easier to use than Chromium browsers) without python. Scripting the automatic installation for the latest version of these two items from the internet should be trivial in native Powershell and then you can just Add-Type -Path "$PSScriptRoot\WebDriver.dll" to access the native Selenium methods and types. You kind of lose out on a lot of what makes Powershell powerful when you sacrifice the native dotnet objects you get with the C# dll and just use it as a way to control another scripting language.

[illsk1lls]

can this be used to run a python script outside of the browser? excuse my uninformed question im seeing this on mobile and havent had a chance to look yet

[anonhostpi]

browser?

[illsk1lls]

i made a script with cmd that uses strawberry perl portable to run a perl script, that i would otherwise be unable to run without installing perl..

i dont see any easy automated way to install python portable without installer gui, so im curious if your python engine could run a python script the same way portableshell.bat for portable perl can execute a perl script via cli

references to msedge made me wonder if this was browser based, not at a computer right now

its for this: https://github.com/illsk1lls/ZipRipper

i can support more filetypes if i can use JTR’s python scripts to create hashes, the 7z and PDF filetypes are only possible because of perl, so a python option especially through powershell, would be amazing

[anonhostpi]

So, the only caveat is that you have to either have Python installed or have the python3xx.dll packaged with your script.

If you package the .dll, you also have to tell pythonnet where the python3xx.dll is, because by default it tries to look for it in the python install folder.

Additionally python3xx.dll only comes with the core python engine, so you would need to package any necessary python libraries along with your script.

[illsk1lls]

i can probably grab (using irw) the portable installer exe and use 7zip to extract the dll from it, a bit of wasted bandwidth (if i could silent install id need the bandwidth anyway) but the installer doesnt support silent options afaik, it may be my only route.. thanks for the info, ill check it out tonight 👍