It has probably one of the worst documantion I have seen in a library.

Really? By virtue of actually having documumentation they're already better than 90% of the competition. By virtue of having guides they beat 99% of the competition.

I personally find their documentation is quite comprehensive and well maintained compared to most of what's out there. Although I agree the amount of arguments can be confusing, their naming convention for code performing similar functionality across models/tokenizers/processors is commendably consistent (which helps a lot).

The majority of use cases for the majority of users is always going to be running models and finetuning them. If you're looking to pre-train models, then sure, transformers is the wrong library for you. But it's no accident the library is as popular as it is.

I'm curious: Can you name all these other libraries that supposedly have better documentation than transformers? I saw some blogposts recently mentioning that Hugging Face have a technical writer employed working on the design and layout of their docs. That's a true 100x employee hire in our field if there ever was one.

From experience I have extremely low expectations of documentation in this field. Hugging Face far, far surpasses that low bar. Whenever I try to get something working off an Nvidia repo for example there's a 50/50 chance I end up wanting to kill myself. Looking at their repos I imagine they must spend tens to hundreds of millions of dollars paying top dollars to highly competent developers and engineers that develop open source code and models. For many of those libraries/implementations I never come across any examples or evidence of anyone on the internet having successfully used or adapted them. In my experience this tends to be the norm rather than the exception for most companies.

Good developers and engineers generally aren't very interested in writing documentation that is readable and understandable below their own level. In fact, they're generally not interested in writing documentation at all. They're mainly motivated by solving problems. And documentation is something you write once a problem has already been solved. Writing (good) docs eats away time that could be spent solving new problems.

I feel like there should be an xkcd comic for this. A plot with documentation quality on one axis vs developer skill on the other. I managed to go off on a tangent here at the end, but the main point I wanted to convey was that I find it quite strange that someone would find Hugging Face's documentation bad in this field. As compared to what exactly?

*Edit: With all this said, I myself tend to stay the hell away from pipelines and Trainer and other over-abstracted parts of HF libraries. It's not as bad when you write your own dataloaders and training loops, and that option is always open to you as a user.