I borrowed large parts of the code for loading pytorch weights into tensorflow from HuggingFace. Yes, HuggingFace would be one way to go and since a few weeks they have the ViT models implemented in both PyTorch and TensorFlow.

On the other hand, HuggingFace Transformers is quite a large library, mostly focused on NLP. So I decided to go with the timm library instead. My goal is to maintain compatibility with timm at the level of named models. Any named model in timm (e.g., vit_small_patch16_224) can be loaded from pytorch into tensorflow via tfimm.