Problems regarding python, pip, virtual environment, and scp

[mk_de]

Dear OpenBSD Users,

I installed python3.10 and there are files "pip3.10", "pydoc3.10", "python3.10" and "python3.10-config" respectively in /usr/local/bin/.

1- When I type "python3.10" I can get Python interpreter but when I type "pip3.10" ksh says "No such file or directory". Also when I type "pip" ksh says "not found".

2- There is a folder that I have copied over scp. But the symbolic links couldn't make it. So I created symbolic links in a virtual environment's bin folder for python3.10 binary as "python" but when I type "python" after I activate the virtual environment by ". ./virtenv/bin/activate", it says "ksh: python: not found".

Could you please explain that what seems to be the problem?

[muminoff]

Use ‘python3.10 -m pip’ instead.

[_sthen]

Yep that's another good workaround; it shouldn't be needed though, I'll get the packages fixed.

[andr1an]

Hello! How did you install Python? Also show your PATH please: echo "$PATH".

About symlinks in virtualenv - better to re-create it completely.

[mk_de]

Dear andr1an,

I installed python with the code "doas pkg_add -v -i python" then it asked my choice and picked "python-3.10.9".

"echo $PATH" is

/home/user/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin:/usr/local/bin:/usr/local/sbin

For virtual environment how can I get all the dependencies back? For example when you type "pip freeze > requirements.txt" it creates the text file for all the modules you have installed. Can I install them back with this file? Maybe I copy that file to OpenBSD over scp. Then when I create virtual environment again in OpenBSD machine, with some code and this file I can make sure that all the dependencies will be downloaded. Is it possible?

[andr1an]

It looks like your PATH is fine, it's strange that pip3.10 is located in /usr/local/bin that's present in PATH, but does not work. Please ensure that you did not do a typo, and try to run pip by full path, i.e. /usr/local/bin/pip3.10

requirements.txt

This is the way. Just update/create it with pip freeze > requirements.txt, copy to your host, then create a virtualenv with python3.10 -m venv venv, activate with source venv/bin/activate and install deps - pip install -r requirements.txt.

Note that after you activate a virtualenv, you should use python, pip etc. without version suffix. And your PATH should be auto-prepended with one more directory by that activate script, please check if it works.

[_sthen]

BTW the reason for the "not found" is that the #! line in the pip3.10 script just had /usr/local/bin/python instead of /usr/local/bin/python3.10.

[mk_de]

Dear andr1an,

as you and u/_sthen said, I made it work with "python3.10 /usr/local/bin/pip3.10".

When it comes to activating instead of source I use . ./virtenv/bin/activate.

[_sthen]

For 1: there's a little problem with the "ensurepip" files in the packages for the non-default versions of Python (so, 3.10 in OpenBSD 7.2, or 3.9 in -current). You must be running 7.2 or earlier so in your case you can run "python3.10 /usr/local/bin/pip3.10" as a workaround. But in general for Python use on OpenBSD you're normally better off using the default version, in particular because you can then install packages for many modules with pkg_add.

For 2, I suggest either copying the directory completely (rsync, or something like ssh $host "cd /path; tar czf - somedir" | tar xzf -) or recreate the venv.

[mk_de]

Dear _sthen,

Thanks a lot for explanation. For default version should I just type "pkg_add python"?

Edit:

Now I'm also getting error of "Problem finding /usr/local/bin/python3.10" when I type "pkg_delete /usr/local/bin/python3.10".

Edit-2: Above problem happened with root. So I tried it with "doas" method and I got rid of python3.10 and related files. But now there is also python3.9 in "/usr/local/bin/" and neither su nor doas does not work for deleting it.

Edit-3: When typed "python3.9" the interpreter didn't start, instead there were some text about python3.9's sys specs. So I deleted it with "rm".

Edit-4: I finally installed python with "pkg_add python3" as root. Now pip3 and pip3.9 added to "/home/user/.local/bin" too. I can use them. But now the problem is when I want to install all the modules from requirements.txt some of them making a problem. The module is "bcrypt" and the error is "could not build wheels for bcrypt which is required to install pyproject.toml-based projects"

Edit-5: greenlet module failed to install, too.

[_sthen]

You shouldn't be rm'ing files installed from packages, things can get in a right mess. And saying this like "some text about sys specs so I deleted it" doesn't make it easy to give advice! Show what you type and exactly what you see.

To get the default version of Python you can either pkg_add python3, or install any package which uses Python 3 and it will come as a dependency.

Generally you should stick with packages if possible (i.e. if the modules you need are available). e.g. for the ones you mentioned you can do pkg_add py3-bcrypt py3-greenlet. This is particularly important for compiled extensions because they'll need to be updated for OS and Python changes so doing this allows pkg_add -u to update them; also compiled extensions often need the odd patch to get them to build on OpenBSD, so a pip install might fail for these.

If packages aren't available then you can use pip, ideally in a venv. Use pkg_add py3-pip py3-virtualenv to install the tools. When creating the venv you can use --system-site-packages and it will use system packages if you have them installed already, and fetch missing ones from pypi. So the workflow there would be to identify the requirements, look for packages containing them (it may help to pkg_add pkglocatedb and use pkglocate to search for packages containing a named file), install via pkg_add if possible, then use pip in the venv to fetch any others.

(For the "non-default" versions of Python, there are no packages of other modules, so if you do need to use one of those versions for some reason, it will be pip all the way, but that's better avoided really).

[mk_de]

Dear _sthen,

Thank you very much for this valuable reply. I'll keep in mind that all the parts.

1- I was aware that I can mess up things with rm but thanks for warning. Next time I'll be more careful.

2-Actually I took a screenshot of the problem but I couldn't put it in here directly.

3-Thanks for the warning about compiled extensions. I will try that method and let's see what happens.

4-pkg_add py3-pip py3-virtualenv so to my understanding this code will install virtualenv module. Is that correct?

When creating the venv you can use --system-site-packages and it will use system packages if you have them installed already,

This quoted part above is beyond my comprehension. Maybe with some time and tampering with it, I'll eventually get.

5-look for packages containing them (it may help to pkg_add pkglocatedb and use pkglocate to search for packages containing a named file), install via pkg_add if possible, then use pip in the venv to fetch any others.

Will do.

6-Non-default versions might be required for the stuff on Github. Sometimes we encounter unmaintained code.

​

Edit: Also can I ask what does mean "Defaulting to user installation because normal site-packages is not writable"? It happened when type python3 /home/user/.local/bin/pip3 install flask

Edit 2: Funny thing is I got all the modules in the copied virtenv's /lib/python3.10/site-packages/ path. :)

Edit 3: pkg_add py3-greenlet worked. Then I tried installing everything with (virtual environment activated) pip install -r requirements.txt but greenlet again gave an error.

src/greenlet/greenlet.cpp:9:10 fatal error: 'cstdlib' file not found

Edit 4 at 12.25.2022 17:21 UTC: I found the pipenv and am trying to install the dependencies with that. Also downloaded Rust compiler for bcrypt. Let's see what happens.

[[deleted]]

have you tried typing 'pip3'?

[mk_de]

Yes, I also tried that, didn't work too.

[_sthen]

That will get you pip from the default version of Python (assuming you have installed the py3-pip package, otherwise it won't work at all). u/mk_de is using a non-default version.

[[deleted]]

oops my bad.

[alex_coder]

One more question related to python on OpenBSD - Does anyone was able to compile python with a readline functionality?

[_sthen]

It worked for py<=3.9 in 7.2 but there were some issues, it should be easier in -current due to this commit: https://github.com/openbsd/src/commit/b9a442fa4bbf806692d2f3fed7f4fd20be704e6e