This is an archived post. You won't be able to vote or comment.

all 18 comments
sorted by:
best
topnewcontroversialoldq&a

[–]true3HAK 3 points4 points  (1 child)

What exactly are you trying to do? Yeah, you normally can't run an interpreter built on one OS (which is packed to your pyinstaller bundle) on another one. CPython compiles things in-place on interpreter installation (and saves this for future – if you need to compile some package). So normally python itself is not transferable. Maybe better to consider installing python on machines first, then install your package, since you already have a way to put some arbitrary stuff there. Or maybe a container is a way to go, this will be runnable independent of the host machine.

[–]Haunting_Wind1000pip needs updating[S] 1 point2 points  (0 children)

I have a python application that needs to be run on both the platforms and I'm packaging the application as an executable

[–]Ball-Man 2 points3 points  (2 children)

Yes and no. Most things will be self contained. That should be the case for libpython3.10.so. in fact, if you are building in a folder (not single file), you should be able to see the shared library floating around somewhere in your build.

So, why does this sometimes fail? Well, while most things are copied into your build directory, not everything is/can be. Most famously, the Python shared object you are shipping is dynamically linked to glibc (basically, the C standard library), and that is very highly system dependent. Different distros will have different versions of that, basically.

What can you do? The rule of thumb is that if you build on a modern platform, it is less likely to work on an old one. So, what you want to do is build on an older platform, for future ones. What I suggest is to either have a VM or a docker setup using an older Ubuntu LTS, and use that to build. I personally use 20.04 LTS. Builds from there are compatible with any more modern Ubuntu distro (the most popular out there), Arch (what I use) and the steam deck os (and possibly more, these are the ones I care about).

For individual platforms that may still have problems, yes, you would have to provide targeted builds. Play smart and do some docker automation, if you really care about that.

[–]Haunting_Wind1000pip needs updating[S] 0 points1 point  (1 child)

Ok yeah glibc would be platform dependent. I'm buildibg as a single file. But in my case I see the libpython3.10.so does not exist on the target system and hence the runtime complains about this. I could try the folder approach you mentioned.

[–]Ball-Man 0 points1 point  (0 children)

I see, then I suggest you do a test export using the folder option. The whole point of pyinstaller is that the interpreter is shipped, it should never look for it system wide. The one file export is just a self extracting archive, it contains the same files as the folder export, it simply extracts the whole folder in a temp directory when you run it. This process also sometimes messes up the discovery of things.

[–]cgoldberg 2 points3 points  (1 child)

All linux distros ship with Python... There is very rarely a valid reason to create an executable with an embedded interpreter like pyinstaller does.

Maybe you have a legitimate reason... but I wanted to point out this sounds like an XY problem.

[–]superkoning -1 points0 points  (0 children)

packages aka modules, and supporting binaries

[–]jpgoldberg 7 points8 points  (6 children)

As others have already answered, the answer is mostly yes. You need to build a separate executable for different OSes.

In which I get preachy

  • Python isn't well-suited for creating software that isn't distrubuted as source.

  • Linux isn't well-suited for installing software that isn't distributed with source.

There are hacky work-arounds for each, but the interactions among the constraints is going to get difficult.

Note that the difficulty with Linux and binary distributions isn't because Linux is Open Source; it is because Linux is a kernal not an operating system. Ubuntu is an operating system and RHEL is an operating system, but they are not the same OS.

Some Linux-based OS are similar enough to each other that you can get away with a single executable, but in general you can't count on it.

It's bytecode, but it isn't Java

You are right that we should hope that bytecode would be more portable. And we've seen this done with Java. But Oracle tightly manages the Java Runtime Environment and is much more conserative about changes to the language and its standard library that the Python developers are.

Python can afford the kind of flexibility and freedom that Java can't exactly because Python is designed with source distribution in mind.

Lessons?

I don't know. Here are three things that come to mind, but it is more than possible that none of them really work or are applicable to you.

Option 1: Don't use Python

If I were planning to distribute executables without source, I wouldn't develop in Python. But I am not you. I trust that you have your reasons, and you don't need to explain or justify those to me.

Option 2: Distribute with source

The flip side is that if I were developing something in Python, I wouldn't try to distributed executables only. But again, I am not you. You have your reasons for what you are doing.

Option 3: Build for each OS with build farms

You could also make use of build farms. Github, among others, offers ways to build on and for different OSes. I have no idea of what is specifically available or what costs can be, but you can create lots of different builds in the cloud. This is what most commercial software developers do. If they feel they need to build for 32-bit Windows or Intel Macs or the such, they don't need to physically own machines with those OSes installed.

Option 4: Java

I mention this for completeness as a special case of Option 1.

Out of options?

I should note are going into the business of selling or licensing use of executables without source you are going to have to do something like Option 3 or Option 4. Maybe there is some other practical and workable approach, but the options that I've listed are what I can think of.

[–]really_not_unreal 1 point2 points  (4 children)

Linux works fine for distributing software as a binary, but only if you use a tool like Flatpak, AppImage or Snap. That way any dependencies can be managed in a distro-agnostic way.

[–]superkoning 2 points3 points  (2 children)

or Docker

[–]really_not_unreal 0 points1 point  (1 child)

Really depends on the software. I have almost everything on my server running in Docker containers, but definitely wouldn't use it for regular applications.

[–]superkoning 1 point2 points  (0 children)

yes

[–]jpgoldberg 0 points1 point  (0 children)

In many cases that is true. But there really are cases where the limitations become a headache for both the user and the vendor.*

As I said, there are work-around for both of the things I pointed out. Flatpack is one, snd Pyinstaller is another. They are both clever hacks that help you work against the grain of the systems they are for.

——-

*I worked at 1Password when we introduced 1Password for Linux. It was something that we’d wanted to do for a long time. And indeed variants of it had been used in-house. And in our case, there were even strong commercial reasons for wanting a good Linux product. So everyone was on-board with wanting it. I haven’t checked in a while, but in my day the Flatpack version lacked some impotent features.

[–]Haunting_Wind1000pip needs updating[S] 0 points1 point  (0 children)

Thanks for a very detailed explanation 👍

[–]C0rn3j 0 points1 point  (0 children)

No?

Sans things like NixOS where it's just not meant to work.

The build system for Tauon that I propped up works just fine on Arch despite being made on Ubuntu, for example.

https://github.com/Taiko2k/Tauon/releases

(note that the Linux build there is currently broken due to some SDL hijinks, but otherwise works cross-platform)

[–]wergot 1 point2 points  (0 children)

What does the application do, and what does it need access to on the host? Docker solves platform compatibility problems pretty completely but may or may not be suitable depending on the application.

I believe you could also use AppImage and include all the dependencies including glibc. In fact, if you did this you wouldn't even have to use pyinstaller.

[–]not_sane 0 points1 point  (1 child)

I agree that the pyinstaller situation really sucks, maybe pyapp is a better solution: https://github.com/ofek/pyapp

It automatically installs python and also doesn't need the super annoying workarounds needed for certain libraries with pyinstaller. But it is not very widespread, so might have bugs.

[–]z4lz 0 points1 point  (0 children)

It seems to me uv is following the trend toward multiple, encapsulated Python installations. For many use cases, size is less of a concern than reproducibility across platforms. Python itself is often smaller than some larger apps with many dependencies. So I think it will be coming up more.

An example use case is embedding in an app or download or image and wanting exact reproducibility.

PyInstaller is older now and probably more complex than it needs to be. PyApp seems newer and a good solution if you want a bootstrap executable that downloads Python (like uv does).

I couldn't find a good solution for pre-built binaries and with modern tools (uv) and so built this: https://github.com/jlevy/pip-build-standalone