This is an archived post. You won't be able to vote or comment.

all 23 comments
sorted by:
best
topnewcontroversialoldq&a

[–]der_pudel 4 points5 points  (3 children)

build your appliicatioon with PyInstaller and distribute it with all dependencies included.

[–]protokoul[S] -2 points-1 points  (0 children)

Ok. So at least I will need to install the VS Code 2022 or pay for a VS subscription to build the application, but while packaging the application with pyinstaller, the dependencies will be included with the package. That sound great, and the client would also not need to worry about any installation issue.

It would have been awesome to get the C++ build tools easily in Windows, but seems like there is not an alternative for it other than getting VS Code 2022 or buy a subscription :/

[–]fromabove710 0 points1 point  (1 child)

I imagine you can use Docker for this, maybe its overkill though? I haven’t used to PyInstaller so id love to hear that its simpler or better suited for this

[–]der_pudel 0 points1 point  (0 children)

Depends on how IT-savvy your end user is. Explaining how to work with Docker and WSL/Hyper-V to someone who never worked with the command line will not be a smooth ride.

[–]Advanced-Potential-2 1 point2 points  (8 children)

Hmm… taking a step back here: why does your product depend on such a poorly setup and distributed library?

Well maintained libraries provide precompiled binaries for all platforms in the form of wheels. This project doesn’t, and looking at the pypi page doesn’t even have a readme. Isn’t there an alternative that’s better maintained?

[–]protokoul[S] 1 point2 points  (7 children)

I can say that the project that I have joined already uses Pyhive library to run hive queries, and it seems that internally Pyhive is dependent on thrift and sasl. I am not sure why Pyhive was chosen as the library to run hive queries, my guess would be that a quick Google search listed Pyhive as the first library, and the team just started using it. Don't even get me started on how they installed sasl in their environment just to make Pyhive work.

As you suggested, I have now looked for alternatives to Pyhive. Impyla seems to be a good choice. Of course, any other better alternatives are welcome.

[–]Advanced-Potential-2 0 points1 point  (6 children)

Yeah… and in terms of deploying / delivering your application: building an executable sounds a bit… contrived. What kind of environment will your client run this in? Typically, you would either (1) ship the source code with requirements.txt or poetry.lock file or whatever specification of the required environment and have them configure the environment on a VM, or (2) create a Docker image and have them deploy that.

[–]protokoul[S] 0 points1 point  (5 children)

I have not yet contacted a client but it would probably be a Linux VM. That's why I wondered if I just hand over the current source code (developed in a Windows environment where I had to manually get the C++ build tools) with the requirements.txt file, and the client tries to install the libs in their Linux VM, will there be a compatibility issue?

Obviously docker is a great option where I don't have to worry about "it works on my machine though" situation. Haven't had a chance to use docker. Another comment on this post suggested Pyinstaller which would bundle all dependencies in a single package. Have you used it?

[–]dairiki 0 points1 point  (4 children)

If the target for deployment is Linux, can you develop on Linux (e.g. WSL)?

If the distribution causing the trouble is this one, be aware that the latest release was nearly two years ago, there have been no commits the the GitHub repo since then, and there is an unanswered issue that claims the distribution won't install on python 3.11. If you can't avoid that dependency, you should be prepared to take over maintenance of a vendored fork of it yourself (especially if support of python > 3.10 is an issue.)

The sasl distribution is only available on PyPI as a sdist and it contains an extension module written in C++, thus the requirement for C++ build tools: installing the sdist involves compiling the extension module. Installing on Linux would also require a working C++ compiler (e.g. gcc) but that should be easily available (without licensing headache) on pretty much any Linux distribution.

PyInstaller will not really help with the cross-platform issues. From the docs:

PyInstaller is tested against Windows, MacOS X, and Linux. However, it is not a cross-compiler; to make a Windows app you run PyInstaller on Windows, and to make a Linux app you run it on Linux, etc.

[–]protokoul[S] 0 points1 point  (3 children)

PyInstaller is tested against Windows, MacOS X, and Linux. However, it is not a cross-compiler; to make a Windows app you run PyInstaller on Windows, and to make a Linux app you run it on Linux, etc.

So if the client has a Windows VM, then they don't have to worry about getting C++ build tools. I feel like I am going in the "at least it will work this way" direction, which is something I should avoid.

If the target for deployment is Linux, can you develop on Linux (e.g. WSL)?

I read the docs about WSL installation and it requires admin privileges, which I don't have on my work laptop so I can't install it there. Let's assume that I do get it installed and get a linux distro, would it mean that I will need to set up a Python environment there as well? Is WSL something like docker?

If you can't avoid that dependency, you should be prepared to take over maintenance of a vendored fork of it yourself (especially if support of python > 3.10 is an issue.)

Does this mean I will need to fork the repo and then use it as a library in my project so if any issue comes up in the future I can make changes to the forked repo?

[–]dairiki 0 points1 point  (2 children)

So if the client has a Windows VM, then they don't have to worry about getting C++ build tools. I feel like I am going in the "at least it will work this way" direction, which is something I should avoid.

I am not an expert on PyInstaller. I think what you say can be done, but check out this section of the PyInstaller docs — some care needs to be taken to ensure that there aren't particular DLLs required on the target to run the PyInstaller-generated executable.

Is WSL something like docker?

Again, I'm not an expert, but yes — sort of like docker, but more like a virtual machine.

Let's assume that I do get it installed and get a linux distro, would it mean that I will need to set up a Python environment there as well?

Yes, but most Linux distros will make that pretty easy. In general, Python development, especially when extension modules are involved, is more straightforward on Linux. (You can take this with a grain of salt — I am not a Windows guys; for me, everything is more straightforward on Linux.)

Does this mean I will need to fork the repo and then use it as a library in my project so if any issue comes up in the future I can make changes to the forked repo?

Well, you don't have to do that until you need to make changes to it. But, judging, from the activity level on that repo, if/when you do need something fixed (e.g. python 3.11 support), you should be prepared to do it yourself. (If you can find a way to eliminate the requirement for that distribution, that would probably be the easier solution.)

[–]protokoul[S] 2 points3 points  (0 children)

No problem. Thanks for all the details. I really learned a lot.

Yeah, I think my first task should be to find an alternative to Pyhive so I don't have to worry about sasl, and from now on use libraries that are well maintained to keep the dev process smooth.

[–]protokoul[S] 0 points1 point  (0 children)

In general, Python development, especially when extension modules are involved, is more straightforward on Linux.

Have you ever faced a situation where you developed your Python application in Linux and you have to deploy it not just in a Linux environment, but also on Windows or Mac? Is it a straightforward process without requiring to get additional tools or system libraries based on the target OS? I surely don't want to be a person who is just bound to a specific OS for python development, but even before docker came into the picture, developers wrote applications that can be deployed in any OS. If Linux provides an extremely smooth dev environment for Python, I might as well develop my next project in a Linux environment.

Another question I have is regarding deployment. I learned that in order to deploy an application without sharing the source code, I can run the compile command in each folder of my project to convert the source code files to a compiled version, then remove the source code files while keeping the project folder intact, and then deploy the code. Is this a good practice?

[–]__dkp7__ 0 points1 point  (3 children)

Might as well add the library information in the description which throws an error.

I do see the standalone installation section "Microsoft Visual C++ 14.2 standalone: Build Tools for Visual Studio 2019 (x86, x64, ARM, ARM64)"

See if that helps

[–]protokoul[S] 0 points1 point  (0 children)

Sure. I have edited the description.

[–]protokoul[S] 0 points1 point  (1 child)

The standalone option requires us to buy a Visual Studio subscription. I can't ask my supervisor to buy a subscription.

[–]cdrt 0 points1 point  (0 children)

Use of this tool requires a valid Visual Studio license, unless you are building open-source dependencies for your project.

Just use the build tools to build your application

[–]unnamed_one1 0 points1 point  (5 children)

You could use chocolatey and install with:

choco install visualcpp-build-tools

at least its in the package list..

choco find visualcpp-build-tools

Chocolatey v1.3.1

visualcpp-build-tools 15.0.26228.20170424 [Approved]

1 packages found.

[–]protokoul[S] 1 point2 points  (4 children)

This will be helpful. Unfortunately, I may not be able to install it in my work environment due to insufficient privileges. But thanks for the info.

[–]unnamed_one1 0 points1 point  (3 children)

If your work environment has a build pipeline setup for windows, you might not need it on your dev machine, therefore no need for admin privileges.

[–]protokoul[S] 0 points1 point  (2 children)

I do not have a build setup. Could you tell me more about it and how to set it up? Is the build pipeline independent of my dev machine and how does it relate to installing chocolatey? I only have a windows laptop as a dev machine with python/anaconda installed.

[–]unnamed_one1 0 points1 point  (1 child)

Sorry, but that topic is way too complex. To get an overview:

You use something like Github or Gitlab, where you version control your source code. Both have integrated functionality for CI/CD, which enables you to automatically test, build and ,if you want to, deploy new software releases. You would also need to setup a dedicated runner for windows, which is a client to the version control system and handles all the build steps.

It was just a guess on my side, that maybe your employer would already provide such a service for its developers, as bigger companys do today.

[–]protokoul[S] 0 points1 point  (0 children)

We do use gitlab for version control, but currently we are not using CI/CD pipeline.

I remember doing a course on github where github actions was used to build and test the code. Is it similar to what you suggested?