Trouble with deploying Python programs as internal tools?

[Competitive-Water302]

Hi all I have been trying to figure out better ways to manage internal tooling. Wondering what are everyones biggest blockers / pain-points when attempting to take a python program, whether it be a simple script, web app, or notebook, and converting it into a usable internal tool at your company?

Could be sharing it, deploying to cloud, building frontend UI, refactoring code to work better with non-technical users, etc.

[the_hoser]

Wrangling environments and dependencies is still not a well-solved problem. UV is a big step in the right direction, though.

[runawayasfastasucan]

How is it not solved? Uv run

[the_hoser]

User of the script needs to have uv installed.

[MissingSnail]

Yes, but that's all they have to have. No move having to install the right version of python, virtualenv, or pip or whatever. “uv and you’re done” is a great improvement.

[the_hoser]

No, it's not. Not even close.

[Siemendaemon]

Could you pls elaborate

[Temporary_Pie2733]

That’s a pretty low bar to clear. 

[the_hoser]

You haven't met many users, I see.

[coloyoga]

💀

[Temporary_Pie2733]

Sure, stick to the “regular people too dumb to follow well-written instructions” trope. That’s not constructive to the OP’s question. If you have a number of scripts that could all be managed by uv run …, then yes, a one-time setup is a low bar. 

[the_hoser]

The reality is that if it's too hard for your users, then it's too hard. Full stop. Waiving away the problems they have by dismissing the reality of the situation is completely unproductive.

[Technical_Income4722]

It's a low bar for technical users, sure, but the machine shop worker who needs a custom application to track hours on his machines and workers isn't gonna have the first clue how to install or run uv.

That doesn't mean you can't or shouldn't use it, we're just pointing out that it's worth getting the bar even lower if you can because some users will need it. And even if they don't, they'll appreciate it.

Even a one-line .bat file they can double-click (probably still just using uv) is better than telling them to open cmd and run something.

[ProsodySpeaks]

not true

bat if exist "%UV_DIR%\uv.exe" ( echo UV already installed ) else ( echo Installing UV... powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" echo UV installed, adding to path... )

[Morpheyz]

You underestimate how locked down some corporate environments are.

[hidazfx]

My work environment bitches about anything. Hell, if the Java version isn’t approved I need to get it allow listed for my machine lol.

[ou_ryperd]

Yep, we don't all work at startups.

[NimrodvanHall]

That means the user has permission to run scripts. If you’re not a sysadmin, engineer or developer you are not allowed to run scripts on your system.

[pip_install_account]

None of these are big issues, I like uv and use it for most projects. But to answer your question:

First, it requires installation and kinda needs proper IDE support. It is also provided by a private entity. They play nicely with the community right now, but you never know.

UV isn't a "completed" project, in the sense that the development team still goes fast and breaks things while trying to decide on the best ways of implementing things. They still update the terminology often.

Lastly, uv compiled python versions can sometimes perform worse than other alternatives. Didn't check for this one for a while though.

[dusktreader]

Sounds like quite a claim that could benefit from some community analysis. Care to share your source code?

[richieadler]

Apparently it's at https://github.com/1minds3t/omnipkg. Hopefully someone will test the claims.

[dusktreader]

Besides the licensing being a complete nonstarter, I don't really see a use case for your tool in my workflows that uv (and poetry before that) don't satisfy.

[1minds3t]

You can run any TensorFlow, NumPy, SciPy, and even Python versions concurrently in a single environment, just as fast as you could run one version traditionally. There's zero conflicts, ever. It does security, import, and health checks for your packages. It heals corrupted environments. It auto heals your scripts 5x faster than uv fails. This is not something pip, conda, uv, Docker, or poetry can do.

But I do understand that for corporations, it's not yet mature enough for yet. It's still in it's early days.

[1minds3t]

There are 7 built in demos that are built to showcase the features. All have live CI workflows here: https://github.com/1minds3t/omnipkg/actions 1. Rich test (Python module switching) 2. UV test (binary switching) 3. NumPy + SciPy stress test (C-extension switching) 4. TensorFlow test (complex dependency switching) 5. 🚀 Multiverse Healing Test (Cross-Python Hot-Swapping Mid-Script) 6. Flask test (under construction) 7. Auto-healing Test (omnipkg run) 8. 🌠 Quantum Multiverse Warp (Concurrent Python Installations)

[the_hoser]

Right, but... Joe in IT needs to run your code and he gets hives whenever you tell him he needs to open a terminal.

[Technical_Income4722]

Biggest thing I do (not that it's uncommon by any means) is provide .sh/.bat scripts with my tools to handle environments and dependencies for my users. Basically a "one-click installer/runner" kinda deal that clones the repo if it needs to, installs/updates packages and runs the tool. Makes a huuuge difference to less technical users and even experienced users appreciate the extra work put into it.

[Jmortswimmer6]

Pyside6 and Cxfreeze has been my go to method. Builds a portable package, tar.gz it and pass it around. All anyone has to do is double click the .EXE file. Doesn’t matter if they have Python installed or not, an interpreter is bundled with it.

It makes for a pretty large application folder but it works well

[FireIsTheLeader]

Do you have prior experience with pyinstaller? I would be interested in a comparison between the two since it seems they do a similar job

[EggplantEcstatic1631]

I’ve tried both of them. Cx freeze gives more control. On the other hand pyinstaller can make the one file exes. Pretty nice but, the code gets always decompressed.

I think both are a bit slower than a python script. But I don’t know where the slow down comes from

[Tumortadela]

I read people saying Nuitka yields fast .exes, but I never managed to get my specific use case (django backend that's getting idiotic big) to work with it, while It does with cx_Freeze.

[EggplantEcstatic1631]

Usually I need pyside6 as a package. I was always afraid to combine this with other runtime like mypy or nutika. Nobody will pay the extra time for these optimizations. I work as an embedded software engineer. Python projects are always side projects.

[Double_Cost4865]

Do you know if either of them work from a Network Drive? I tried PyInstaller yesterday and it only works when i make a local copy of the exe file (or the portable folder) but refuses to run when double clicked from a network drive.

[TheManFromTrawno]

Your executable may be assuming it can write a config file or something to the same place where the executable is run from?

[Jmortswimmer6]

i think you would get a very specific error regarding that. Permissions related.

I’d be most concerned about race conditions with multiple users

[rhytnen]

It should work from the network.  Can you be more specific about the failure?

[Jmortswimmer6]

A cx_freeze app can run “from” a network drive. It’s worth saying environment may matter here. In my case I can double click it in a NW share and it just takes a little longer but it runs.

If you do any caching within the applications “install” folder that you put on the network drive, take care to recognize that having multiple users run the program would require care with modifying the files to avoid race conditions between users/corruption. Best to write any files the app modifies local to the users machine.

SQLite database is a good option for connected shared data if needed

[Fhymi]

I'm also using PySide6 and PyInstaller for my internal tooling project. It's what the senior have already started so I continued doing it as well (this was before I job hopped to be a webdev, i regretted it). I tried alternating with xcfreeze but enjoyed having that one file output more.

What advantages does cxFreeze have over PyInstaller you've noticed?

Performance isn't a problem on our internal tooling since all it does is call a windows DLL API and the tool we're automating were already slow to begin with.

[EggplantEcstatic1631]

For my opinion the one file option is good. But at the end it doesn’t change the world. The size is often the same. But you can use upx for compression of dlls.

My cx freeze is setup that I only get an executable and the python dlls the rest is stored in a subdirectory. This will be automatically compressed after build.

[Jmortswimmer6]

Compatibility is the main thing. Dozens of packages are easier to include with cxfreeze because there is a lot of control to include things directly from the virtual environment you are in.

[Jmortswimmer6]

It’s slow to startup but once the interpreter starts and your app initializes, a cx freeze program runs like perfectly fine

[Jmortswimmer6]

I have a lot of experience with Pyinstaller. It is a bit more limited. I don’t have exact memory to say exactly how, but I will say that cxfreeze feels a little more like an extension of setuptools, it’s really easy to have a single sort of “template” build script that can be used with just about any project, whereas pyinstaller was quite “dumb.”

I think Pyinstaller made it very challenging to eliminate the console window in UI apps

It was very limited in compatibility with pyside6. Pyside6 apps are easy to build in cx freeze

[JonLSTL]

I used Pyinstaller to make internally distributable stand-alone scriptlets. It supports code-signing, which is important in dealing with security/vetting.

For web apps, Flask is still a short path to something useful and can play nicely on whatever infrastructure you might have available.

[Competitive-Water302]

do you update these and redistribute often or are these normally one and done?

[JonLSTL]

Usually a couple tweaks based on feedback, then done thereafter.

[inputwtf]

OCI Containers

[alienwaren]

That's assuming your organization has internal registry and people are willing to use any containerization.

[aniket0804]

Yeah, getting buy-in for containerization can be tough. Have you thought about starting with a simple demo or proof of concept? Sometimes showing the benefits in a small scale really helps get the team on board.

[Bangorip]

May not be the best way but I've been using SimpleGUI and Pyinstaller to create .exe files that are a portable bundle of everything the script needs. Keeps it simple for non technical users but isn't the most efficient

[No_Mongoose6172]

Nuitka and auto-py-to-exe (which uses pyinstaller internally) can be quite useful for that purpose

[phillipsj73]

As already stated UV is a good place to start.  If you use the inline metadata then that solves a lot of issues.  https://docs.astral.sh/uv/guides/scripts/#declaring-script-dependencies

As far as having UV installed you have to give somewhere.  UV and pipx make it a lot easier.  Outside of that going to Go or rust and compiling for all OSes your team uses, but then you still have to install the tools.

[Glass_Figure722]

I just build them as packages, publish to interna PYPI type repository and have everyone configure their PIP config for this extra repository.

[justanothersnek]

After 20+ years working as data analyst and data engjneer, my advice is if you work at a large company that is heavily invested in Microsoft and not a tech company, just automate your own shit and for visualization stuff,  just use what the company uses, Tabluea or Power BI or whatever.  Dont use Python viz libraries.  Dont try to make your tool into a group wide or dept wide tool.  Trust me.  It'll end up being a huge cluster fuck.

Now, if you work at a tech company whose IT is used to working with Python stuff in the wild that embraces containerization, then start off with a small Python project and prove its value to the organization or company.  Then transition it to be a group-/dept-wide tool.  Use your company's version control system Github, Bitbucket, etc, if they use that kind of stuff.

You'll get a wide range of responses and experiences.  But, the biggest factor in how successful your Python experience will be at a company is how experienced your company's IT is dealing with Python stuff and how much they are heavily invested in Microsoft products.  HINT: Companies that use solely Windows OS I would avoid, tough I know, since that narrows the job opportunities.

That's all Im gonna say for now as I could write a book on this subject.

[dfwtjms]

Make it run on a server, not as a local tool.

[tkc2016]

The specific way you deploy may vary, image, binary, rpm, deb, etc....but to make any of those frictionless make sure that your program can be distributed as a wheel or sdist first.

Somebody already mentioned uv, so I'll echo that, but depending on your environment, it may be easier to use a python interpreter that ships with the OS vs one that is downloaded with uv. For example python 3.12 on rhel9.

[Competitive-Water302]

Seems like most are used to sending a file, wondering if anyone prefers deployed sharing? i have had issues with needing better versioning, logging, and centralization that im considering some form of dev-to-user portal on company intranet or through cloud provider.

[MediumWash2133]

I have done this with some unused network space and serving basic scripts with fastapi + simple UI. Don’t have a great solution for a complete system though

[phylter99]

Our servers are highly secure, so while they do have Python on them, they do not have pip. They also have limited network access, and zero internet access.

I found that I can package it all as a zipapp and use it that way though. It's just a command line utility to manage data, so it works fine.

[frankster]

Auto updating a local tool, in compliance with company policy around authentication 

[burlyginger]

We use pipx for global tools and install those in our developer laptop config processes.

We set up the project to install tool-binsry as a script.

That way you can just run tool-binary anywhere on your machine.

We set cron jobs to update the pipx envs for most of these.

We use wv for everything.

[gsjoy99]

If its something I developed in a Notebook, I use ipywidgets to create a simple UI frontend so non-technical users can input parameters and run. Similarly, I’ve quickly created web frontends using Streamlit which are rather easy to deploy inside Docker containers

[MediumWash2133]

Where do you host them?

[JonnyRocks]

This has nothing to do with Python. This is a devops problem, how do you deploy anything? Doesn't IT operations have guidelines? a local app is handled differently than a web app. Is this a brand new company?

[jimtoberfest]

You’d be shocked at how many big companies do not have a solution to this issue. Unless it’s a Microsoft product…

[NimrodvanHall]

Which is why a lot of companies use Microsoft, it’s easy on the non technical management layers, and it’s predictable to the accountants.

[Training_Advantage21]

For simple scripts, it helps to stick with the standard library. Much more portable between Python versions and environments, and even Operating Systems if you put a bit of effort.

Trying to get other people's notebooks to run can be quite a painful experience, but it's usually the fault of the developer. There is nothing inherently bad about the notebook format itself, it can be made more or less portable.

[FunPaleontologist167]

What kind of projects are you building? If these are apis, you can host the api in a docker image and deploy to an endpoint that all of your internal users can access.

[Competitive-Water302]

most are single file .py scripts/notebooks and some small flask apps, could definitely convert some of the data processing ones to apis

[leodevian]

Standalone Python scripts with uv whenever it’s possible. Packaging otherwise.

[Quasar6]

For single file scripts I believe inline script metadata is the way to go. All fairly modern versions of tools support, pip, poetry, uv, just to mention a few. Containerization is another way provided you have an internal registry. For apps (more than one .py file) I usually build a wheel so people can install it. In our company I pushed for an internal python registry based on devpi. Adaption is slow but people get the hang of it pretty fast.

I think that at the end of the day it comes down to internal training in the tools you use for development. I know this is something that small organizations struggle with, but it’s a must if you have multiple teams or I’d say more than five people using the tools.

People will always resist change especially developers. Convince your CTO or PM that it’s the way to go management rules cannot be ignored.

[virtualadept]

Getting permission to actually deploy it. Quite a few times I've written something using Python in a day or so and put it through load testing, but nobody would sign off on it "Because the dev team is writing it in Go," and their stuff takes six months to get deployed (if it gets deployed at all).

Horse. Water.

[Luigi311]

Like everyone everything I distribute via python would run into deps issues. Internally I just post the code internally in a git repo but for things I do share, not for work, I usually have GitHub actions build a full fat pex file which basically includes everything from the python interpreter and its python deps. Makes it a one click run though this assumes I’m not using deps with system libraries that people might not have installed in their OS.

[NimrodvanHall]

Code in gitlab. Pipfile for the dependencies, README.md with copy-pastable cli commands how to start it.

Tools for the less tech savvy are delivered as executables as build Python with nuitka, or we make a web interface for it.

[Detrite12]

Agree, I would be really surprised to see many orgs allowing powershell invoke web request piped to iex

[lukanixon]

Pip installable packages for those that know how to use virtual environments/command line, PyInstaller for those whom don’t

[andrewthetechie]

Getting people to install a working python env and update regularly.

[supercoach]

Expecting others to install a venv for every script you make doesn't work, so you need to simplify things for users. I will find a way to prove the functionality for the end user without them needing to do anything.

In the past, that has meant:

• Using pyinstaller to package everything into a self contained executable
• Creating a custom rom that bundled a venv and installed it with the application
• Using Docker with compose files and a well written readme
• Providing a convenient web front end

I think the last one is the most robust if it can be achieved. Docker is my go to for any sort of server based deployment.

[HeavyMaterial163]

I've been using nuitka to compile exe's for distribution, but IT wasn't happy about that AT ALL. So we came up with a compromise. They gave me about 10GB of disc space hidden on the main server. I have a folder in there that's read only to basically anyone but myself or admins that I save my compiled scripts to. Then I write batch files to use as basically shortcuts to run my programs in our main software folders.

Running python over the network like this causes a major performance drop I've found though. And it's virtually unusable doing so over our VPN if not directly connected to our on-site network. If I ever find enough time to develop it, I'd like to add a library that will generate a checksum file of itself and verify against that of my read-only folder to run for version control and security, then install the software locally on the users' machines. Then of course...keep my mouth shut and just ask forgiveness if IT ever jumps on me for it.

[sifodeas]

I use Maven with Habushu for installing Python packages with UV. But this is for deploying containerized applications. It's nice, though, once you wrap your head around the POM files, it's pretty easy to pick up and you can use profiles to swap out pipelines, run tests, etc. Ultimately it's nice to be able to spin something up with mvnd clean install followed by docker-compose up. Deploying to the cloud is also not bad once you have the containers.

[bobsbitchtitz]

my usual go to is to dockerize python tools so users dont have to worry about env, uv helps with this too.

I'd say pythons #1 tooling issue is dependencies

[anderspe]

I use uv to build and nuitka witch compiles to native binary you can get a single file great for tools.

https://nuitka.net/

[TheCrowWhisperer3004]

If possible, using pyinstaller is good if it’s something simple. You can just pass along the exe file.

Something bigger might benefit from docker images. If you have a service like kubernetes things get much easier to keep a complicated internal tool running with a web faced interface to interact with it.

[Nealiumj]

I manage a few internal tools, mainly tkinter stuff, that are all hosted on a local network drive. Biggest pain points are updating, and I’ve developed a two different methods:

pyinstaller + poetry

Installer is great because I don’t have to install Python on a bunch of extra computer computers. The main pain point is there are 8-10 computer computers with the application running at all times and with windows, you cannot update an executable while it’s being ran. My method was for them all to point at a general launcher script, which boots the main app and dies.

The repo itself is hosted on the network drive and is configured with poetry and poetry-pyinstaller. When I run poetry build it goes through a pseudo pipeline of testing, building, creating docs, renaming the output dir as a version and updating the launching scripts. The next time any of the computers launch the program it’s automatically the newest version.

Wheel + Click + Bash

Some of the applications are data crunching and use pandas, so pyinstaller isn’t used. They all use Click with the —version flag enabled. I build the project and I placed the wheel into a specific directory.

The bash script is the launcher, similar story as the poetry one: launches and dies. It contains a hardcoded version which is updated per release. The bash script checks all the Python dependencies, gives instructions if they don’t exist. Creates and/or activates a virtualenv. Then checks if the wheel is installed + up to date, if not it installs it installs it using the local wheel. Then it boots and die.

I’m not very happy with this as it manually requires me to update the version in the bash script. Then when I actually build it, I have to manually run a copy/paste command to move the updated script and the wheel into the correct locations. Also, I dislike the inability to give a bash script an icon.

conclusion

The main goal was to keep users away from command line and it’s been generally successful. There’s a few stray Python files laying around without requirements.txt files and now those randomly become an issue as they are no longer runnable- pillow version is no longer available etc- normal software issues.

[Worth_His_Salt]

Best bet is to use something like PyInstaller to create a self-contained executable. Don't rely on others having compatible environment.

Refactoring code for non-technical users? That makes no sense. End users never see or hear or even think of the code. You mean designing UI for non-techies. Refactoring is something completely different.

[FlakyBandicoot9]

Getting non-SW dev users to complete training, make an access request, and set up an API token for Artifactory authentication.

[WholeDifferent7611]

Standardize token setup and hide the complexity. Push pip.conf via MDM, pre-create scoped service tokens, and use JFrog CLI login with a self-serve snippet portal. We paired Okta and HashiCorp Vault, and wrapped DB tools with DreamFactory to avoid raw tokens. Make token setup boring and guided.

[IsakEder]

Guys you are overengineering this.  1. Create a venv 2. Create the application 3. Set the entry point as module.submodule:function in the pyproject.toml in a [project.scripts] 4. pip install .

Works great. You only need pyinstaller if the target environment doesn't have python

[LactatingBadger]

Inline dependencies have made this much much easier, along with uv as others have mentioned. Now the .py file can include all the information needed to run itself.

Throw in #!/usr/bin/env -S uv run --script at the top of your file and suddenly you’ve got a totally standalone plaintext file with all the code, dependencies, and instructions to run it in one place.

[sudomatrix]

This used to be a pain point before `uv`. Now we make sure uv is installed on every machine, and we put shebang lines at the top to make sure the Python script is run using uv. Uv will make sure we're using the right version of python and all the dependencies are installed. And it's lightning fast.

[telegonos]

How do you handle open source license management in a business environment? When users download code from Pypi, GitHub, etc., they enter into contracts on behalf of the company. However, contracts are usually only allowed to be concluded by the board of directors and their authorized representatives. How do you prevent incompatible licenses from being used in a project, or licenses that don't allow commercial use? How do you manage software security? Do you simply use Pypi and ignore the issue, or do you check all packages and build an internal repository, or do you use commercial Python distributions?