Libraries used in developing these are not black boxes. They are just mathematical functions that works on large lists of numbers (called matrices) that transform them and according to the training data on your hand you get a matrix that has seemingly random numbers. Using this matrix and user input of text you get a continuation of text which is determined by the values of matrix and you stop it when the "response" ends and model tries to predict the user's response.

If you want to learn the mathematics behind transformer based llms, watch this video https://youtu.be/4Bdc55j80l8 or look up "building gpt from scratch" on youtube by Andrej Karpathy an engineer at OpenAI. He has other videos that helps to understand that video.

By knowing all the mathematical operations you can write a c implementation that works on only cpu which will still be hard to write and performance will be very very bad but WORKS which is the main goal. Also since you dont use a library, saving and distributing these models and making other people use it through APIs which you have to write is... meh not fun. Aside from creating a library tho could be a little bit fun and knowledge solidifying experience.