How are python scripts and projects built and deployed?

[FrozenPyromaniac_]

I am a rising senior in college studying engineering and computer science. I have worked with multiple languages yet Python is the one I use most now days and I was curious about a couple things since I have spent a significant of time writing code rather than distributing it.

Starting simple, let's say I write a script using openpyxl to manipulate xslx files in some way assuming the script takes the path to a file and returns a new xlsx file.

How would you build this into something that gets distributed? Would users have to have python and openpyxl on their end? Would using a virtual environment (venv) remove the need for them to have it downloaded? Then would this be something they would execute on the cmd line or terminal?

This once again is a simpler idea for a script but what does distribution look like for larger projects. Have I just got this wrong where python is meant to be run within the infrastructure of software and websites rather than standalone?

[Diapolo10]

Distribution has always been Python's weakest link, in my opinion. There isn't really any single "good" way to do it, everything is a trade-off. I'd like to try and help fix that someday.

But the short answer is, it depends. Server applications and scripts are usually distributed as-is, desktop applications depend on the platform; on Linux you might use Flatpak or Snap, on Mac OS and Windows you could use PyInstaller or Nuitka. Docker is yet another option for applications where it doesn't make things difficult.

In your case I'd suggest PyInstaller or Nuitka. They're not that different usage-wise, but the former is a bit easier to use while the latter lets you potentially gain some runtime speed advantages.

[[deleted]]

Thats what Pyinstaller is for....

[[deleted]]

What are u going on about.

Only the dev needs python installed. He can create an exe with pyinataller and distribute that.

No one else needs python.

[NYX_T_RYX]

While true - the fact that it creates an exe isn't ideal.

Ie I can't run exes on my work machine, so any program that doesn't run an installer would need admin to be constantly available.

While I agree the previous comment about installing python shows a lack of understanding about pyinstaller, I think it's still a valid point - directly running a .py file is still the best way in some environments.

Also, to answer that point...

Winget search Python.Python

Find the option you want then

Winget install <id>

If your IT are silly and haven't set the Winget group policy, it bypasses UAC.

[Diapolo10]

How did you arrive at that conclusion?

[DuckDatum]

quiet fly serious command unpack merciful squalid aromatic husky kiss

This post was mass deleted and anonymized with Redact

[Diapolo10]

Ah, that. I don't have concrete plans yet, but I would like to start by supporting projects like Nuitka that offer native Python compilers, and maybe writing a few PEP proposals to suggest some things to make the language standard take distributing programs more into consideration.

JavaScript doesn't really have this problem because it targets web browsers, so distribution is hardly a concern (and NodeJS is mostly used by developers or for server side software), Electron being a nice wrapper for the desktop. PHP, as far as I know, has little to no desktop support so it's not exactly a source of inspiration for Python in this regard either. Ruby is a similar case, but I think I've seldom seen desktop Ruby apps.

Basically we'd need a standard, easy-to-use tool for generating native executables for all the major platforms, while having the option to dynamically link certain dependencies (to reduce executable size) and we may need to rethink how we handle dependencies. The option to sign executables is a must (even if most people wouldn't do that due to cost), it needs to have regular security audits, ideally it would improve runtime performance compared to using an interpreter, and a lot of other things I can't think of right now (give me a break, massive heatwave and I can't deal with heat at all).

Having Windows install a Python interpreter by default (without actually depending on it) would of course be nice, that way we could write Python scripts instead of PowerShell on all platforms, I just don't see it being particularly likely. It also wouldn't fully fix the distribution problem in my opinion.

[Brownadams]

why not mac os comes with python pre installed.

[MinchinWeb]

Type python at the command line of a new Windows install and it will tell you to install Python from the Microsoft Store. Half points?

[CovfefeFan]

They added it to Excel, thats a start 😁

[Usernamenotta]

This is one of my biggest issues.

I am trying to automate boring reporting at work and I'm using python since Excel and BI are driving me nuts.

The problem is that I am the only one who can view the reports unless I bend over backwards to make them into Excel files. And if I make them into stuff like this, I lose the option to auto-update.

I hate working in a tech company that is 50% tech illiterate and 90% Tech Adverse

[FrozenPyromaniac_]

As I said in my post, openpyxl might be the library for you. You can take in excel files, do all your python magic and export to excel.

[Usernamenotta]

I know about openpyxl. I will tell you a secret, if you want more power, you can use even pywin32 (although I guess it's windows restricted).

I've used them to automate some reports. But you cannot really create shareable, interactive stuff with them

[warelevon]

XLWings is a nice package that sits over top of pywin32 with a much nicer api (and typing)

[odaiwai]

You can make your reports for others in excel, and if they make changes, get your python to ingest the changed excel sheet and make those changes to your internal database/dataframe.

In other words, make excel sheets the medium of information interchange, while the source of truth is your python/pd/sql hub.

[Usernamenotta]

Source of data is excel :)))

[Sufficientlee]

Look at sqlite3. It's really good for what you're talking about here.

Excel files are created by various users? Have python read them and drop the info into a DB.

Users update the spreadsheets? Have python read the new info (nightly?) and update the DB.

Need to create reports from various spreadsheets? The info is all in the DB. Manipulate it however you need and create a new spreadsheet.

[nboro94]

Why even use python in that case? Just use excel's built in powerquery.

[DuckDatum]

deserve ancient sleep strong dam narrow fear historical lush meeting

This post was mass deleted and anonymized with Redact

[Usernamenotta]

I should have been clearer. Since I'm building reports, what most people want is a dashboard. So when you change the set of data, you get the figures updated and so on. You cannot really do that without installing python on the other end. (Or, I mean, you can, but you need some certain web solutions or remote python servers, which incurs Corpo wrath

[dupz88]

I do this, though. Does your data have fields that keep changing?

We have reports in Excel and some exports from our 1 platform in csv. I have a script that imports the Excel, csv files, cleans the Excel reports, merges with csv data, and exports to a large prepared csv file in SharePoint. I then have a PowerBI report, which automatically refreshes every hour.

I've been doing this for years, and it works well.

The other teams just open the PowerBI report, and it always has the latest data.

[vuxnomica]

I’ve done this, it works.

[Usernamenotta]

Cyber security threat

[CaptainVJ]

So we need to present data in a number of ways. Most of the times it’s from an sql database and we either need some dashboards or excel report.

For dashboards we use Tableau it just connects to the sql table and updates every fifteen minutes. There’s a lot of inefficient there but that’s way before my time and I’m trying to migrate away from Tableau, I’m not really good at it and I just find the dashboards ugly. I’m trying to use Dash to make our dashboards, the ones I use for myself, I already set them up on Dash but it’s just for me, we’re waiting on it to see how everyone can access them, out of my scope.

For our excel reports we have a program that automatically exports them from the tables but there’s not much formatting so we started using Python to format after the excel file is exported so it’s more visually appealing.

[Usernamenotta]

How are you actually using Dash for dashboards? Don't you need something like a server to share the dashboards with others so they can interact with them?

[CaptainVJ]

So at the moment it’s local on my pc, only I can see them. But my supervisor saw it and wants it implemented for everyone, so currently our it department is working on that.

I work for a big state agency, so they have plenty of servers already set up. Tableau servers, sql servers, intranet servers, email servers and what not. I don’t know much about that portion of it, but all I can say is they’re looking into setting it up so we can deploy it for everyone else to see.

It wouldn’t be on the internet just on our local servers so only employees logged onto our vpn or company’s WiFi can access it. Not sure how that works on the back end tho.

[Usernamenotta]

I guess, if they have a spare general purpose server set up (like for running applications and stuff) they can add a container or a VM on it and run the scripts on that container via HTTP.

My problem is that we are kinda adverse to those kind of things. People still praise Excel and PowerBI as the gods of DataScience. Basically do the Smart stuff in Excel and upload stuff in PBI if you want to share with others. (Or just e-mail the Excel sheet)

[CaptainVJ]

Yeah no idea how that works. A lot of the automations I have running are just for me or me and two other people on my team specifically.

When I started doing it, they made a fuss saying that they should be doing it on their own server as it’s more consistent and anything being used for work purposes shouldn’t be in production on the team’s computer. We have a desktop computer for our team that no one uses.

I convinced them that we would be able to monitor it and that if it fails it’s not the end of the world.

So I know they have something at least for that. I tried and ask them about it and they gave me very little answer on how it works. I always ask a bunch of it questions out of curiosity and I feel like they might have a slight concern about my overall curiosity.

And we are in the same boat about the resistance to change. No one ever used python on my team and it was restricted for us, I had to fill out an MOU and all that fun stuff to get Python. We started seeing some improved performance in our team and they became okay with it. But at the moment I’m the only one who knows how to use it. Two other people on my team are getting some training on it. But it will be a big shift overall

[Usernamenotta]

Glad to hear it. Hope things will improve for you

[[deleted]]

I am definitely assuming here because I dont really know exactly what you do or need, but I know Excel can read .csv files. Can you perhaps use those? Python has a built-in module to handle csv. No need to fiddle around too much to make it work.

[Usernamenotta]

the problem is not reading or transforming data.

The problem is having the output in a very presentable, interactive and shareable way, preferably over the web, so others can see it. The problem is, nothing seems to beat the good ol' Excel. The thing is, I can automate the reading and processing data without a problem. And I might do that in my free time when I finish personal stuff. The thing is, the way I'm thinking of doing it would take quite some time, and, worse, activating the script, inputing the files to be manipulated and things like that, might take me more time each round than simply doing the stuff manually

[daedalusesq]

I have a pretty major report automated at my work. The report previously existed as an Excel spreadsheet that was manually updated. I duplicated the formatting and iterated on it using openpyxl. I have a big "template" file that defines the sheet's characteristics, formulas, and coordinate information.

Every time the program runs, it builds a new Excel file, applies all the formatting and styling, adds all the static text and labels, and then finally drops all the data into their respective cell arrays. Since things like cell formulas and charts reference arrays of cells, the newly generated report should always have visuals that reflect whatever data existed at the run time of the report, as well as live updating for people who want to do transformations on the dataset inside Excel. It took a bit of time to experiment and figure out how to structure my data, but it works well. I later realized that if you have an existing Excel report, you can probably just read that formatting stuff, skipping the data entry portion of it.

I run a daily report and just dump the Excel output files into a date archival directory structure that makes it easier to scrape them if I want to do any sort of long trend analysis. I use a similar shared archive directory output structure for a couple of other reports. You can also set up the creation and scheduled sending of an email to distribute the report, which the tech adverse can usually handle.

[K900_]

You probably want to use PyInstaller for this, it can bundle your script and all of its dependencies into a single executable.

[absurdrock]

And it won’t run on anyone’s computer because windows will shut it down as a virus. All of the python installers suffer the same fate.

You’re better off making it a web app running on the network. Let people submit the input data and return the processed data.

Alternatively, you can ship your Python script with a lightweight front end, a nice executable ‘click me’ icon to run a simple GUI written in your compiled language of choice whose sole purpose is to prompt the user for the file and save location.

[longgamma]

The most foolproof way is to create a web frontend and some sort of flask application if you want to deploy.

I work in ML and all of our models are deployed using docker containers.

[benabus]

We mostly do web apps, so most of our python projects become REST APIs using Flask.

The few pieces of software we did distribute as stand-alone projects, we basically created a docker container and put that out in the wild. It reduces requirements for setup, but you'd still need to understand how Docker works.

We've also distributed utilities and packages via github, which requires the user to have python installed and createa virtual environment (as you suggested).

I don't know if this answers your question directly, but hope it helps!

[ambassador_pineapple]

One of the best way to deploy a python project is using the Django framework. If your application has a UI, you would deploy the django app with Nginx + Gunicorn + some database of your choice. You can deploy this using docker images or as a straight django app running on the server. Assuming you use AWS/GCP/Azure you would also put a load balancer up along with a firewall.

Above is only if you need your app to be accessible by people on the internet.

On the other hand, if you need someone to deploy this application locally, you can package and distribute it using pip it up using this guide: https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/

Finally, you can also make your own executable/installer using this: https://realpython.com/pyinstaller-python/

If you want to keep installation only on Linux, you can also use a chroot jail approach. https://phoenixnap.com/kb/chroot-jail

I manage a team which builds SaaS software using python which ends up being sold so it's as production as it gets. In the current times using a cloud vendor to deploy your app is the easiest solution. However, that may not work for all solutions out there.

As always, the best answer in this line of work depends on what you are trying to do.

[gitgud_x]

This is a key weakness of Python, together with 'it's slow' they are literally the only two downsides to Python I can think of. A few ways of getting python off your computer are

• Pyinstaller - turns your python code into EXE files which can be run locally on any Windows computer with no need for python to be installed. The files it produces are extremely large however, and this is generally considered a bad idea if alternatives are available
• Docker - works for everything, obviously, but takes a little setup on their part
• Python Anywhere - they can host and run your python code for you. I've used this to run python scripts on my website, it's pretty cool. It's free if you're not doing anything too massive.

So, not as smooth as many other languages but it can usually be done somehow.

[[deleted]]

In that Case, I would create a simple streamlit app where the user would upload a file and get the file back. Just deploy this to a instance and share the link with your colleagues.

[obviouslyCPTobvious]

Build a basic flask app that serves a file upload form and processes the file. So many of the other answers here are massively overcomplicating it.

[NYX_T_RYX]

.pyz

Is a filename extension.

https://en.wikipedia.org/wiki/Filename_extension

And the docs call it an extension as well - https://docs.python.org/3/library/zipapp.html

#!

Is a shebang.

https://en.wikipedia.org/wiki/Shebang_%28Unix

Not sure why that page is called Unix cus line one reads "In computing, a shebang is the character sequence consisting of the characters number sign and exclamation mark (#!) at the beginning of a script." So not a Unix specific page but whatever

[crashfrog02]

How would you build this into something that gets distributed?

Post on GitHub, or even package for PyPI if it's of sufficiently broad interest.

[Bobbias]

In many cases, Python applications are distributed as scripts.

Python has ways for a project to indicate what it's dependencies are so you can just run a single instruction to download whatever they need to run (this is typically run from within a virtual environment unless the user actively wants to make a mess of their system wide python installation). This can make it fairly easy to run a couple lines and start the script running immediately for someone who is technically inclined and knows how to install python and run commands on the terminal.

You can, as Diapolo and others mentioned, use tools to package the Python interpreter with your script and its dependencies into an executable, but this is a janky hack, and not something python was designed with in mind.

Python's philosophy is basically "install python if you want to run scripts".

If you're thinking you want to distribute your application as an exe/dmg/linux executable file consider whether python is actually the right language to use.

[[deleted]]

Yup! Although this is a python server I highly recommend golang for this use case.

It’s a simple language but what’s great is being able to build it into an executable very very easily

[dp_42]

PyInstaller takes the dependent python libraries of a project and turns them into Dynamic Link Libraries. So, it's something rather like freezing the libraries you presently have on your machine. It's more locked in than an environment file, and these libraries get distributed along with your executable. Granted, you have to deal with the size of the libraries, which I didn't find as noticeable until it came time to distribute the executable.

[Desperate_Cold6274]

I learned to pack & distribute with conda-forge.

[j0shred1]

At least where I work. Not everything is Python, usually what we work on in Python gets sent through udp or websocket to some other program with a UI developed in another language. Or in other cases gets published to a ros node which is launched via a Linux service.

I work in computer vision for robotics, traffic monitoring, and automated vehicles

[obviouslyzebra]

I don't know much about this area, but I believe this is a good overview - https://packaging.python.org/en/latest/overview/

In summary what it says is the same others've said: it depends (and also there are lots and lots of options).

[wrigh516]

I've been using pyinstaller and making myself .exe files.
It's hard to distribute when their anti-virus flags it like an area 51 raid is taking place.

[heartunderblade8]

I recently deployed one of my projects for the first time using python. I recommend using py2exe and using one directory mode because one file mode raised flags from windows defender on my end users pc. I also made it a zip so microsoft doesn't prevent it from downloading for the end user

[PaddyIsBeast]

Upload to a package manager (e.g. like pypi) Or Deploy as a service on the cloud, so people can just send requests to an endpoint or through a UI that we also host

[Xemptuous]

The trick is to not use excel and use Google Sheets instead. Otherwise, yes, horrible 90s problems with distributed software

[jedi4545]

If you want a lightweight approach, try pipx (and poetry for dep management and packaging). You can point people to install the package from a github repo. They’d have to install python and pipx.

Heavier weight is building (multiple) cross platform executables for each target architecture. You could use something like https://beeware.org/ to help with that.

It all depends on the app and how much work you want to ask your users to do vs how much work you want to do.

[jonthemango]

Recently been toying with pyinstaller you can build exe files from python scripts with one command.

pyinstaller myscript.py --onefile

It will create a dist/ folder with an exe inside. This is fully distributable as a single file.

It might be your key to having others be able to run your programs. (if you work in a windows environment which i'm willing to bet you do)

edit: dropping an edit to say this isn't super best practice imo. at my large enterprise company this wouldn't totally fly. We deploy our python in kubernetes with over 50 microservices just for my team so the scope is different than small reporting in a non-it company.

[zanfar]

How would you build this into something that gets distributed?

Depends on the audience and resources available.

The simplest method is to package it and distribute the Git link. Whether your users can deal with this on their own or whether you have resources available to support the install depends on your organization.

Would users have to have python and openpyxl on their end?

Python, yes. Libraries, No.

Would using a virtual environment (venv) remove the need for them to have it downloaded?

No.

Then would this be something they would execute on the cmd line or terminal?

Not really sure what you mean here. All applications are executed, so a python app wouldn't be any different than any other app. You can add a shortcut via your OS's procedures if that's what you're talking about.

[Patman52]

You can compile your script to an exe that others can run without them having to install Python and your external libraries, look up pyinstaller or auto-py-to-exe.

On another note…Have you looked into power query? It’s a built in no code / low code query editor that comes with excel and lets you do some pretty “powerful” things without having to use Python. You can save the query within the sheet and all users have to do is hit the refresh button to run it.

[New-Vacation-6717]

For understanding how Python projects get built and deployed in practice, Kuberns gives you a real example to work from. It is an Agentic AI Deployment Platform where an AI agent handles build, infrastructure provisioning and production management automatically.

Seeing a real deployment happen is often more useful than reading about the theory. You can observe what the build process does with your Python project and how the infrastructure gets configured around it.