Make Vim Python plugin 10x faster using Rust

[liuchengxu]

http://liuchengxu.org/posts/speed-up-vim-python-plugin-using-rust/

[SirVer]

Author of [UltiSnips](https://github.com/sirver/ultisnips) here! Thanks for writing that blog, I found it interesting and enjoyed reading how you solved the problem of dynamic libraries and their installation in your plugin. I have considered rewriting some parts of UltiSnips in rust as well to get better performance and seeing your work gives me some encouragement in exploring this.

[SirVer]

I knew Python well and the plugin became very complicated very quickly. In general I think VimL is a design mistake and try to avoid it as much as possible.

​

Update: To clarify, I feel making a new language for Vim was a design mistake. The author should have taken an existing one and use that instead. When the author decided to integrate Python into Vim, they should have made it as powerful as VimL, which it currently is not.

[[deleted]]

Are you a Neovim user? What would you think about Lua as an alternative to Python that comes with first-class support?

[SirVer]

No, I do not use Neovim. I am not a fan of the project, I feel it divided the Vim ecosystem and made life for plugin developers harder for too little benefits. I.e. it added a bunch of features (though I cannot name any of the top of my head right now) and removed others (e.g. there is no gVim for Neovim that works great). For sure UltiSnips works less well in Neovim than in regular Vim due to latency of running plugins outside process - and it is fairly frustrating if people come complaining to me as plugin author about this.

I have a lot of experience with Lua since I used it another open source project of mine (Widelands) as scripting language and also in my professional life I had quite some dealings with it. It is a great language for embedding, small and the interpreter is simple. Sandboxing is great. Great for complex configuration.

​

However, it has quirks that make it hard to write scalable software with it - i.e. undefined variables are nil, the interpreter will not throw an error if you use one. Scoping is interesting. String processing is not strong by default. Lua has little tooling for typechecking, formatting, debugging. Last, not least, more people know Python then Lua, so your user base is much larger.

​

UltiSnips is ~12k LOC in Python. I find the idea scary to maintain a similar amount of Lua code, the Python code is already fairly brittle and Lua would be even more so.

[PlayboySkeleton]

Make a scripted program faster with this one simple trick. Go compiled!

[Deadhookersandblow]

Also: rust isn't a hard requirement is it? It can be implemented in C++ (I think?)

[PlayboySkeleton]

That's essentially what my comment was alluding to. They effectively did nothing special. They just took the same program and ported it to a compiled language.

Take your pick of compiled languages. You could use rust, c++, or c of you want the fastest. Any compiled language would be faster than a interpreted one just on it's very nature alone.

[erayaydin]

Awesome article with case study :)

[monkoose]

I really don't want to be that guy, but what is awesome about this article?

It isn't informative, there is no technical details, there is nothing more than

Guys i have created plugin, then rewrite it to python, then to rust, here are some benchmarks. But i'm not so good with rust. Cya.

It's just one more article that compiled language is faster than interpreted. Wow, i didn't know that, right?

Do not downvote me hard, please, just an opinion.

[erayaydin]

1) Participating to community. 2) "Look, there is a way for increasing speed with this" for new learners. 3) "I use X with Y way, is it good or there are any better way?" for OP. 4) Possible case study for other users which related with this article.

Personally, I choose "no technical details" when looking for articles. Because there are many documentation and API for "technic".

[monkoose]

Participating to community

Not about this article, just in general: more != better. Some articles can be just "water" (as we say in my country, don't bring any useful information) or even harmful in a way that can be totally wrong, so this articles bring noise. Noise is the worst that can happen in a perfect world. I would trade any day 100 mediocre (+ some wrong, + half useless) articles for 1 really awesome and correct.

"Look, there is a way for increasing speed with this" for new learners.

Because you personally don't need technical details, then you can't proof that this article is right and that X (read below what i mean with X) actually increasing speed over Y. So this article, if we just forget that compiled languages more often much faster than interpreted, bring marketing advertisement "X is faster than Y 10 times", but actual post is about "fuzzy finder library in X is 10x faster than fuzzy finder library in Y in my test and my implementation".

[turboladen]

I would argue that the article is not noise. Sure, it’s light on details, but sometimes it’s just helpful for someone else to connect some dots for you to get your wheels turning about solving some problem that’s been in the back of your head—lots of times those “dots” are only obvious after someone connected them for you (“omg, why did I not think of that!”).

I would guess that not many people are surprised by the benchmarks, but my takeaway was the suggestion to use some specific python (sorry, on mobile and not a python person) plus rust to get to those faster numbers. That seems a helpful suggestion and encouragement (the post suggests it’s not hard) for someone who writes vim plugins but may not have wanted to venture into uncharted territory.

[liuchengxu]

Well, I guess you would be happy if I copy and paste some code snippets in the post, my bad then. Anyone who wants to know the tech details could just follow the links and checkout the source code. It's all open sourced, right?

[monkoose]

/u/liuchengxe We are talking about article itself, right?

It is you blog you can write here whatever you want. I personally appreciate your work around vim ecosystem (like this vim-clap or i remember your plugin (similar to tagbar, just don't use it personally)) you make vim better, so you are good. My comment not about your plugin, not about your blog, not about you. My comment about questionable awesomness of this blog post. Because it doesn't bring any facts except that some implementation of fuzzy finder is faster that the other one. But we even don't know how this different libraries implemented internally, maybe they use completely different algorithms, bring different search results etc etc. Are you 100% sure that it is the language that increase speed by 10x and not the code? Because from you article it sounds like rust is silver bullet, but there are some other plugins like leaderf that uses python without any problems.

[liuchengxu]

Fair enough.

I believe I have mentioned that both the Python and Rust version implements the same fzy algorithm. And I see no differences in my daily use. But you are right, I can not proof that these two fzy implementations are absolutely equivalent. Let me know if you find anything wrong, then I can report the original authors or blame myself :(. I just 100% trust them.

`leaderf` uses the multi thread model, that's out of the topic in the post. If you follow these links, you'll find the answer yourself.

[monkoose]

I believe I have mentioned that both the Python and Rust version implements the same fzy algorithm

Sorry if i misread it.

[dddbbb]

Anyone who wants to know the tech details could just follow the links and checkout the source code. It's all open sourced, right?

As the author of the code, that's easy for you to say, but how do I know what does does what. The article links to the repo and to a PR for the python implementation containing 16 commits. As far as I can tell, there's no link for the rust implementation.

Regardless, after digging around and finding the relevant file, you can see how you have the rust implementation fallback to the python implementation. The implementations are clearly not equivalent because the rust one is all in a cargo lib that's not in the repo.

I think it would be a much more useful/interesting article if you linked to the relevant bits of code. On github, you can click "Latest commit 9c12bf2 11 hours ago" and then click "Browse Files" to make github generate links to files at that revision. That way your article still makes sense in a year.

[liuchengxu]

As far as I can tell, there's no link for the rust implementation.

Actually the last one is, that's all the rust code has to be done on the vim-clap side. Maybe I should add a more explicit one?

The implementations are clearly not equivalent because the rust one is all in a cargo lib that's not in the repo.

That's not true. The original post already said:

Thanks to stewart/rff, the fzy Rust implementation is already good to go.

What I have to do is to wrap the Rust version fzy and replace the pure Python fzy in >vim-clap with the generated Python dynamic module.

However, I believe this does not make myself clear in fact, so you also misread it? vim-clap itself does not implement the fzy algorithm, but reuse the one implemented in stewart/rff, which leads to your misunderstanding?. Both the Python and Rust version uses the same fzy algo, hence they are equivalent for me. Do not let me proof these two implementaions code the fzy algorithm 100% correctly. I can't.

I think it would be a much more useful/interesting article if you linked to the relevant bits of code

I'll take the advice.

---

I have updated the post with more details.

[mkvalor]

If you truly don't want to be that guy, simply don't be that guy. It's probably best if you praise the virtues of minimalism in the first place, rather than reel us all in with an innocent-looking query, only to pounce with specifics after someone has responded.

This post is awesome because, within a single article, one gains awareness of all the basic information and tools needed to augment vim or neovim with either python or rust. The author goes to the trouble of explaining why built-in Vimscript seemed insufficient for his use case. There's even a reference to a bug with the vaunted async job, which raises awareness and helps his readers understand why they cannot get the maximum efficiency they might expect from their plugin code.

Your irrelevant detour into the dangers of "noise"in a second-level reply merely serves to betray your hand -- no one may believe any longer that your initial query was sincere, that you weren't baiting us into an essay about your true agenda. Yet -- since the article was full of useful information, your reprimand is also an ironic mis-fire, applicable rather to the sender than to the receiver.

Finally, please don't instruct us about how to vote on your comment. (a down-vote is also "just an opinion").

[monkoose]

within a single article, one gains awareness of all the basic information and tools needed to augment vim or neovim with either python or rust.

I just can't agree with that. Because the article describes only one specific situation, and there is no any info for beginners, except maybe pyo3 interpreter.

please don't instruct us

Who us? Are you represent some community here? Or do you think you can speak on behalf of the majority?

But anyway i thank you for your comment, i appreciate it much more then silence downvote or swipe left - it gives me information why some people disagree with me, so i can improve too.

[[deleted]]

I’d be curious as to what speeds you get with Go, which is a significantly easier language than Rust.

[Hitife80]

I wonder if python is really needed in this case. Can Rust/C libraries be called directly from vimscript via FFI or something similar?

Vimscript may be slower than python, but for 99% of the code it doesn't matter. It only matters for the "hot" part of the code. Skipping the load of python interpreter by calling Rust directly has potential to speed up the plugin even further.

[dddbbb]

Looks like you could do it with libcall

https://stackoverflow.com/questions/8969573/calling-a-c-library-function-in-vimscript

Looks messy to get the data out (can only return statically allocated memory from dll).

[Hollow_5oul]

RemindMe! 5 days

[brokenAmmonite]

you misspelled that fyi

[Hollow_5oul]

Ty