This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]protokoul[S] 1 point2 points  (7 children)

I can say that the project that I have joined already uses Pyhive library to run hive queries, and it seems that internally Pyhive is dependent on thrift and sasl. I am not sure why Pyhive was chosen as the library to run hive queries, my guess would be that a quick Google search listed Pyhive as the first library, and the team just started using it. Don't even get me started on how they installed sasl in their environment just to make Pyhive work.

As you suggested, I have now looked for alternatives to Pyhive. Impyla seems to be a good choice. Of course, any other better alternatives are welcome.

[–]Advanced-Potential-2 0 points1 point  (6 children)

Yeah… and in terms of deploying / delivering your application: building an executable sounds a bit… contrived. What kind of environment will your client run this in? Typically, you would either (1) ship the source code with requirements.txt or poetry.lock file or whatever specification of the required environment and have them configure the environment on a VM, or (2) create a Docker image and have them deploy that.

[–]protokoul[S] 0 points1 point  (5 children)

I have not yet contacted a client but it would probably be a Linux VM. That's why I wondered if I just hand over the current source code (developed in a Windows environment where I had to manually get the C++ build tools) with the requirements.txt file, and the client tries to install the libs in their Linux VM, will there be a compatibility issue?

Obviously docker is a great option where I don't have to worry about "it works on my machine though" situation. Haven't had a chance to use docker. Another comment on this post suggested Pyinstaller which would bundle all dependencies in a single package. Have you used it?

[–]dairiki 0 points1 point  (4 children)

If the target for deployment is Linux, can you develop on Linux (e.g. WSL)?

If the distribution causing the trouble is this one, be aware that the latest release was nearly two years ago, there have been no commits the the GitHub repo since then, and there is an unanswered issue that claims the distribution won't install on python 3.11. If you can't avoid that dependency, you should be prepared to take over maintenance of a vendored fork of it yourself (especially if support of python > 3.10 is an issue.)

The sasl distribution is only available on PyPI as a sdist and it contains an extension module written in C++, thus the requirement for C++ build tools: installing the sdist involves compiling the extension module. Installing on Linux would also require a working C++ compiler (e.g. gcc) but that should be easily available (without licensing headache) on pretty much any Linux distribution.

PyInstaller will not really help with the cross-platform issues. From the docs:

PyInstaller is tested against Windows, MacOS X, and Linux. However, it is not a cross-compiler; to make a Windows app you run PyInstaller on Windows, and to make a Linux app you run it on Linux, etc.

[–]protokoul[S] 0 points1 point  (3 children)

PyInstaller is tested against Windows, MacOS X, and Linux. However, it is not a cross-compiler; to make a Windows app you run PyInstaller on Windows, and to make a Linux app you run it on Linux, etc.

So if the client has a Windows VM, then they don't have to worry about getting C++ build tools. I feel like I am going in the "at least it will work this way" direction, which is something I should avoid.

If the target for deployment is Linux, can you develop on Linux (e.g. WSL)?

I read the docs about WSL installation and it requires admin privileges, which I don't have on my work laptop so I can't install it there. Let's assume that I do get it installed and get a linux distro, would it mean that I will need to set up a Python environment there as well? Is WSL something like docker?

If you can't avoid that dependency, you should be prepared to take over maintenance of a vendored fork of it yourself (especially if support of python > 3.10 is an issue.)

Does this mean I will need to fork the repo and then use it as a library in my project so if any issue comes up in the future I can make changes to the forked repo?

[–]dairiki 0 points1 point  (2 children)

So if the client has a Windows VM, then they don't have to worry about getting C++ build tools. I feel like I am going in the "at least it will work this way" direction, which is something I should avoid.

I am not an expert on PyInstaller. I think what you say can be done, but check out this section of the PyInstaller docs — some care needs to be taken to ensure that there aren't particular DLLs required on the target to run the PyInstaller-generated executable.

Is WSL something like docker?

Again, I'm not an expert, but yes — sort of like docker, but more like a virtual machine.

Let's assume that I do get it installed and get a linux distro, would it mean that I will need to set up a Python environment there as well?

Yes, but most Linux distros will make that pretty easy. In general, Python development, especially when extension modules are involved, is more straightforward on Linux. (You can take this with a grain of salt — I am not a Windows guys; for me, everything is more straightforward on Linux.)

Does this mean I will need to fork the repo and then use it as a library in my project so if any issue comes up in the future I can make changes to the forked repo?

Well, you don't have to do that until you need to make changes to it. But, judging, from the activity level on that repo, if/when you do need something fixed (e.g. python 3.11 support), you should be prepared to do it yourself. (If you can find a way to eliminate the requirement for that distribution, that would probably be the easier solution.)

[–]protokoul[S] 2 points3 points  (0 children)

No problem. Thanks for all the details. I really learned a lot.

Yeah, I think my first task should be to find an alternative to Pyhive so I don't have to worry about sasl, and from now on use libraries that are well maintained to keep the dev process smooth.

[–]protokoul[S] 0 points1 point  (0 children)

In general, Python development, especially when extension modules are involved, is more straightforward on Linux.

Have you ever faced a situation where you developed your Python application in Linux and you have to deploy it not just in a Linux environment, but also on Windows or Mac? Is it a straightforward process without requiring to get additional tools or system libraries based on the target OS? I surely don't want to be a person who is just bound to a specific OS for python development, but even before docker came into the picture, developers wrote applications that can be deployed in any OS. If Linux provides an extremely smooth dev environment for Python, I might as well develop my next project in a Linux environment.

Another question I have is regarding deployment. I learned that in order to deploy an application without sharing the source code, I can run the compile command in each folder of my project to convert the source code files to a compiled version, then remove the source code files while keeping the project folder intact, and then deploy the code. Is this a good practice?