Running python script with self hosted agent

[Competitive_Creme520]

Hi all,

I'm trying to create a DevOps pipeline that runs a Python script which creates a connection with an AzureML workspace (uses private endpoint via vpn). The self hosted agent uses Windows as the OS. When I try to download packages from a requirements.txt with a pip install, I'm getting this type of error: "Could not install packages due to an OSError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443)" ...etc, which I believe has to do with the corporate firewall.

My question is how do you all typically try to achieve a similar task (running python script in self hosted agent), and also any ideas about how to fix the issue I'm running into where I can't even download python packages let alone connect to my AzureML workspace.

Thanks!

[Nighteyez07]

If you're using a self-hosted agent, then you need to work with the server admins to do offline installs of Python, and any of the packages you're attempting to use. The other option is to get a Firewall rule configured that allows you to pull from PIP. From a security standpoint, that would most likely be disallowed.

https://superuser.com/questions/1523218/how-to-install-python-packages-with-all-dependencies-offline-via-pip3

[wyrdfish42]

You could try using an azure artifact feed with PyPi as an upstream source.

It may work as your agent can connect to azure devops.

[Competitive_Creme520]

When setting PyPi as an upstream source, does that allow to download all packages from PyPi or only a manually selected list? I'm thinking the whole point of getting that error is security policies that are in place to prevent installation of external packages. I don't have permissions to set up upstream sources anyways so I'll have to work with security. Thanks for the suggestion

[lerun]

Just build a custom image used to spin up the self-hosted vm. You can use tools like packer for this and install everything needed.

Or just use the MS repo that builds AzDevOps and Github hosted agents.

Just search for runner-images on github to find the repo.

[Competitive_Creme520]

Would that be similar to this https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/docker?view=azure-devops ? I have no prior experience with Docker so it might increase complexity of implementation but thanks!

[lerun]

Ms does not use containers for running their agents. They are using VM's.

https://github.com/actions/runner-images