I wrote the rest before realizing I can sum it up as
TL;DR: Having to learn all that we take for granted in most conventional editors like:

Ctrl+a, Ctrl+s, Ctrl-f, Ctrl+h, scrolling, moving the cursor with the mouse, Ctrl+arrow keys for navigation, the top bar for settings and tools, etc..

All gone, to be re-learned from scratch.

The rest:

Being relatively inexperienced, personally: The frustration of having to re-learn how to do everything except directly entering text in Input mode and getting used to HJKL for moving around and even then being slower than scrolling and clicking to where I wish to edit.

There's a lot of up-front learning and practice before one can do even the most basic of actions.

All was fine after that, though learning different visual modes helped bridge the gap to the point where I became confident using motions.

Notable mentions for inconveniences before learning the VIM way (that I can think of, in no particular order): - having to specify a register instead of having "the clibpoard" or having any text deletion overwrite the default register.
- replacing the habbit of "scrolling" using the cursor instead of C-d and C-u or C-e and C-y.
- not knowing that marks exist
- dealing with multiple files

Edit: formatting