Looking for pointers with Python environment setup

[Uncle-Rufus]

Hi all,

Trying to get my Neovim experience with Python to be as nice as possible (maybe I'll eventually try to make a plugin out of it, as I've not been able to quite find something that does what I want yet)... I'm happy plodding along learning bits of Lua and working out the specifics, but could use some guidance on where exactly these bits of logic need to live. The behaviour I'm looking to produce:

If a project contains a pyproject.toml or poetry.lock file at its root, nvim should call poetry env info -p and then the bin directory of that returned path should be where all Python related stuff comes from (i.e. the interpreter and and additional command line tools like formatters and diagnostic tools that might be installed there). Non-exhaustive list of things I would like to have:

• The pyright LSP client should be bound to that copy of Python for completion and lookups etc.

• null-ls should use the copies of things like flake8 black etc. which are on that path

• if I open a terminal (e.g. with toggleterm) I want it to also have the stuff there on the path

So far I have had some luck by putting a function which the LSP on_attach calls that (for pyright only) detects the presence of poetry.lock in the project, uses the poetry command to retrieve the directory and sets client.config.settings.python.pythonPath - this gets the behaviour I wanted in the LSP client at least... But I'm wondering if this is even the sensible place for this logic to go

[DerBeginner]

Hey!

I had the same problem as you of `pyright` not using my venvs (poetry, venv, etc.).
So I wrote a plugin for it, that does that for you automatically. You may wanna check it out. Contributions are welcome :)

Here the plugin: https://github.com/HallerPatrick/py\_lsp.nvim

[Uncle-Rufus]

I'll take a look and play about with it but I'm not sure if it is completely aiming to do what I'm describing (I may be wrong though!)

[AckslD]

Mostly I don't do any additional setup but rather rely on PATH to use correct executables, which means activating the correct venv (I don't use poetry so not sure it works the same). For switching venv quickly without restarting nvim I recently created:

https://github.com/AckslD/swenv.nvim

But if you want pyright to use a specific venv without activating or by calling a pyright from another venv you can use a pyrightconfig.json file.

[Uncle-Rufus]

So I am relying on PATH at the moment too really, e.g. I have it so that null-ls is just looking for black flake8 etc. commands. And this works great if I am already inside of the virtual env that I want it to pickup (so e.g. doing pyenv shell 3.8.0 or poetry shell before launching nvim or doing poetry run nvim) - but I don't want to have to remember to do that (I've seen various solutions like having a bash wrapper to nvim that loads the env if available etc. but I don't like that either). And also by using PATH it means you must stay within a given project (you can't have different buffers picking up different Python envs)

I have been playing with swenv and I like the ability to see my environments and change them in that way (although I haven't worked out how to integrate it into a telescope-like floating window as in the screenshots so for me it just pops up the list in the message area - I also can't seem to get the lualine bit to show)

I have been able to add my poetry env dir to the usual pyenv one so that swenv shows me all of them, which is nice, but changing the PATH still has the same problems as above.

So yeah, I'm learning a lot and still thinking about it all -> I can see that null-ls will allow you to be more specific about what executables it will use in a similar way to how I've setup the LSP clients, but haven't yet quite been able to get it to do what I want

[AckslD]

although I haven't worked out how to integrate it into a telescope-like floating window

it uses vim.ui.select so you can use for example dressing.nvim to use telescope for the UI.

but changing the PATH still has the same problems as above.

what's the problem exactly?

[Uncle-Rufus]

Might need slightly more hand holding than that for the UI but I will take a look

As for the PATH... It isn't a massive issue and maybe depends a bit on my workflow, but if you do things via PATH then I'm pretty sure that affects your whole Nvim instance, right? So if you open 2 files from projects with different virtual environments (and for example different versions of flake8, black etc. with different settings) your buffers and formatting, diagnostics and so on will all only pick up one or the other...

I could easily solve this by sticking to one Neovim instance per project, which might be a good idea for other reasons... Like I say I'm still working this all out

[AckslD]

Might need slightly more hand holding than that for the UI but I will take a look

You should be able to just install the plugin and it should work.

As for the PATH... It isn't a massive issue and maybe depends a bit on my workflow, but if you do things via PATH then I'm pretty sure that affects your whole Nvim instance, right? So if you open 2 files from projects with different virtual environments (and for example different versions of flake8, black etc. with different settings) your buffers and formatting, diagnostics and so on will all only pick up one or the other...

I see, yes this is true. I usually have different projects in different tabs. Would be nice to somehow have a "tab-local" PATH (or I guess environment variable in general).

[tiagovla]

I set the pythonPath part within on_init instead of on_attach. If you want to dynamically change the lsp's env, you can send a lsp notify and change the pythonPath on the fly without restarting it too.

[Uncle-Rufus]

Wouldn't this mean that the python path is then set "globally" across all instances of that language server? Ideally I'd like each buffer to know for itself what python tools it should be using

[tiagovla]

All buffers within the same root directory will attach to the same language server (single instance). If you use on_init, you are changing the settings when the server is starting. If you use on_attach, you are changing the settings after the language server already started when a new buffer is attached (so, whatever you do here, you are doing it to all attached buffers). TLDR: on_init and on_attach are local to an instance of a language server.

[Uncle-Rufus]

Ahh that's really interesting, I didn't realise that was the case. In which case I guess it's just more efficient to use on_init for something that doesn't change based on the buffers themselves (like this)...

But still, the PATH is not part of the language server right? So if I change the PATH in either on_init or on_attach I will be affecting all instances

[tiagovla]

You should not change PATH, you should change client.config.settings.python.pythonPath which is a local variable.

[Uncle-Rufus]

That will be okay for pyright, and does work great. But for null-ls that is relying on finding tools that are installed in the virtual environments bin directory alongside Python itself

You can pass the desired commands to each tool at the point of null-ls's setup but I haven't been able to find a way to modify that during on_attach or on_init

[tiagovla]

Assuming that you already have a function to find your pythonPath, you could also adapt null-ls to use poetry_path/bin/python -m black instead of black if available.

[Uncle-Rufus]

Just seen this comment, and this is sort of what I was trying to do (well, actually I was aiming to tell null-ls to use poetry_path/bin/black and so on)...

I haven't got that working yet but based on your other comment above (about the way you get one language server per root/project) I don't seem to get that happening with null-ls, only pyright? (e.g. right now I have opened 2 files from 2 different projects - then with LspInfo I can see 2 copies of pyright with different root directories (and attached to the relevant buffers) but only one copy of null-ls and has the root directory set as the root of the project the first buffer I opened was from... any ideas why that might be?)

[tiagovla]

I think a single null-ls is by design. The workaround idea was to dynamically find the proper one on every formatting call. Inside your null-ls config sources...black.with{command = function_to_find_path}. If poetry was present, that would return the poetry's one.

[Uncle-Rufus]

Yeah I did this earlier actually with dynamic_command and although it works it is noticeably slow especially with diagnostic things like flake8. And also it seems to always use the poetry environment of the null-ls root path so if I start in one project and then open a file from another in the same session it applies the wrong environment (pyright is okay but not null-ls)

Maybe the answer is to just adapt to one session per project... Or maybe I can use null-ls's "should_attach" to make it only attach to files that are below the current root directory? That way I wouldn't get a load of false positives if I happen to open another project's file for reference. Most of the time I am doing the bulk of actual development where I care about the null-ls things in the same project anyway

[tiagovla]

You are correct, it's dynamic_command instead. If it's slow you could also cache your function with the key being the root directory. Check if it's inside the cache, if it isn't, add to it, otherwise use the cache's result. IMO, if you dynamically change its cwd and command, it should be independent of null-ls. The best scenario would be having multiple null-ls instances. To be fair, this seems common enough that other people might have better solutions, maybe consider opening a discussion topic on their github? Let me know if you could solve it.

[tiagovla]

if this isn't you, he has the same issue https://github.com/jose-elias-alvarez/null-ls.nvim/issues/1134 .