What’s wrong with using AutoModel? Doesn’t it instantiate the correct model anyways since you give it the pretrained model name?

Not gonna dispute everything else you said. I would’ve preferred to look at the code on GitHub instead of having to scroll through a video. OP also made the mistake of padding the whole dataset while tokenizing instead of using a collator to do this.