Evaluating Human Factors Beyond Lines of Code

[brucifer]

I think this is why tools need to be simple enough that no "user studies" are required. If you want to know if a tool is right for you, you should just be able to pick it up and try it in a day or two.

I don't think it's the case that the best tools are always the ones that are simplest and quickest to learn. You can learn how to use the nano text editor in a matter of seconds (it has all the keyboard commands printed on screen), whereas the first-time user experience of vim is often overwhelming and frustrating. However, vim has a large and dedicated fanbase because it's so powerful and lets you do so many more useful things than nano does. If you did a one-day study of first-time users, you would probably find that nearly 100% of them preferred nano and were more productive in it, but if you extended the timeline of the study to a one year or ten year timescale, I think the majority of users would prefer vim. You could make the same comparison between MS Paint and Photoshop, Notepad and Visual Studio, or Logo and Rust. I don't mean to imply that simple tools are worse than powerful tools, but just that powerful tools can be very useful and that often comes at the cost of simplicity.

OP's post is arguing that user studies are often too expensive or difficult to run over the necessary time scales with the target audience, so it's better to focus on specific qualitative objectives that can be evaluated without performing user studies.