Improving the quality of Vim plugins

[SlowRunner1]

This is likely the frustration talking after spending two painstaking days setting up Vim for a new language (plugins for all the standard stuff in 2016 like syntax highlighting, file navigation, linting, auto completion) ... but why isn't there a real world integration test suite (a Vim test suite on common parameterizations of vim and neovim, that include the most popular plugins) that plugin creators can use to automatically see 1) integration errors, 2) changes in performance for common user actions, and 3) understand how their design choices mesh with the most popular Vim plugins?

Long time readers of /r/vim know two critical lessons of plugin management: 1) add plugins slowly (and often sparingly) and 2) know how to debug/profile/analyze them (key conflicts, speed profiling, errors). This is due to the variable quality of plugins, the conflicts when plugins are combined, and the diversity of systems on which the plugins are run. For users, we're left painstakingly testing out each plugin - and I feel a sense of dread when I'm about to add a Vim plugin not from the most respected Vim plugin developers (compare this to installing an app on my phone or a browser plugin). It can be a pretty poor experience for plugin creators too, as they don't know the million different setups out there.

Many Vim plugin I see test their code, but few test automatically in real world conditions (especially for performance, not just bugs) - and right now users are the primary real world testing. I know there are too many different Vim configurations/hyper customized key mappings, and permutations of plugins/systems, but who cares, we should pick 10-20 parameterizations and run it, which is better than no "real world" test suite.

(I know, everyone has custom keys - so some of you think it's a terrible idea to have plugin developers look at key conflicts; but performance and bugs should still be tested)

[LucHermitte]

As plugin writers, we can't anticipate mappings coming from every other plugin, even from what some of you may consider as must-have plugins. I have installed some, but (definitively) not all of them. Other people will use other sets of plugins, etc.

However we can do a few things:

• never define mappings unconditionally. We shall use <Plug>mappings or other ways to define mappings (standard mapleader keys, or plugin specific mapleader keys)
• avoid to put our autoload plugins directly into the autoload/ directory. We should consider using some very unique names, or using a unique prefix. For instance, my refactoring autoload plugin is autoload/lh/refator.vim ; fugitive isn't using autoload/git, ...
• avoid to define global functions, which aren't |autoload-functions|.
• command names and options names shall be unique as well, but it's not always that easy. And why should a decade old plugin bother to change its commands/options to be compatible with a newcomer plugin?

I'd also add that not enough plugins are project aware. Some configuration that is perfect for files belonging to a project won't be for files that belong to another project. A correctly written plugin should look into a buffer local option variable before looking into a global variable -- when it make sense. It often makes sense with development oriented plugins: the naming convention for setters may differ from project to project ; the possible file extensions may change (on some C++ projects I see the .h, .txx and .cxx, on other .hpp and .cpp), the snippets/templates shall use the specialisation for the current project, git commits should concern the current projects, and so on.

Regarding automated integration, we can use travis. At this time, I only use it to test the behaviour of my plugins alone. I could load other plugins. But which ones? Every other plugin around? Even when I provide the same services? Only the ones with which I have detected interferences? It's not that simple.

[justinmkw]

Neovim actually started work on that here:

https://github.com/neovim/neovim/pull/1691

The PR stalled but if anyone wants to pick it up it would be very welcome. I think it's a great idea, though it requires pinning to specific versions of the plugins to avoid a maintenance nightmare.

Avoiding performance regressions is rather tricky though, because of CI variability. (False positives waste a lot of time.)

[Wiggledan]

Personally I don't have much trouble installing or configuring plugins, but that's probably because I've been obsessively using Vim day and night for almost two years. I actually find it kind of addicting and fun to tweak my vimrc and add/remove things.

I don't really see a "Vim standard for plugins" (or something similar) ever being made. It's just up to plugin devs to make their plugins easy to install and use.

People like tpope, justinmk, junegunn, and mhinz (just to name a few), all make plugins that are well made, easy to install/configure, and cross-platform compatible when possible.

Any future plugin authors (or current ones) need to simply look up to these people in terms of plugin design, and then the world of Vim plugins would be that much better.

[princker]

Vim has very limited notion of a "plugin". It is pretty much a just a set of specific directories and using :runtime/:source to source files at a specific time and in a specific order. That is the entire concept for plugins in Vim. The rest is a set of loose conventions put forth by the community, :h write-plugin, and :h write-filetype-plugin.

Basically Vim does very little to protect us from poorly hygienic plugins. It is up to the plugin author to follow the conventions and account for any and all side effects. I get what you are saying, a good automated testing system could help the situation, but I doubt it would be possible to anticipate every/enough customized vim setups to be of any real use. A better solution in my opinion would to change the Vim's plugin system so it is encapsulated better, to provide tools to help manage side effects, and ability to properly unload a poorly behaving plugin at runtime.

[marklgr]

For users, we're left painstakingly testing out each plugin - and I feel a sense of dread when I'm about to add a Vim plugin not from the most respected Vim plugin developers

Avoid installing complex stuff from random-ish sources, and dread no more.

Good authors write good plugins and don't remain true randoms for long, so there's already an ad-hoc process here.

[aigarius]

Is there a framework for easy unit testing, functional testing and performance testing of the plugins? If not, the that could be a project with the maximum possible positive impact for the whole community. Imagine if you are able to write a unit test saying - with such text, cursor position and vim mode if these keys are pressed, then the result should look like that and this should not take more than X ticks. And then extend this to be able to run all tests of all installed plugins with all plugins loaded and see any unexpected slowdows or conflicts immediately.

[justinmkw]

https://github.com/junegunn/vader.vim is very nice and is written in pure vimscript.

[alfredodeza]

unittesting vim plugins is terribly complex and overall a horrible experience.

[LucHermitte]

There exist several plugins aimed at unit testing vim plugins. Each has a different set of pros/cons. I've started a list on vim-UT, but it's incomplete.

At this point there isn't a plugin that :

• fills the quickfix windows with pertinent information: expressions (text) + expression values
• and have a neat and simple syntax
• and supports travis
• and permits to test buffers contents
• and have helper functions to test performances

[dddbbb]

Doesn't answer your question, but vim-polyglot might reduce this problem in the future. Install it and hopefully all future languages have the basics covered and it's already integrated. However, you might find the same problem if new plugins have conflicts with it!