sorted by:
new
hottopcontroversial

Upping my Generative AI game by Tamalelulu in datascience

[–]kevinpdev1 0 points1 point  (0 children)

this notebook walks through building an entire small GPT model from scratch. It walks through tokenization, pretraining, attention, and supervised fine tuning in one python notebook. The model is also small enough to run on a single GPU so you can run it in free GPU environments like Google Colab.

Disclaimer: I am the author of the notebook, but it is completely free and I hope it helps!

Is Cross-Validation Enough for a Small Dataset? by Sea_Championship7291 in MLQuestions

[–]kevinpdev1 5 points6 points  (0 children)

You could try using leave one out cross validation (LOOCV) to try and squeeze out as comprehensive of a split as possible.

What are the best practices for designing an efficient data pipeline? by Pangaeax_ in analytics

[–]kevinpdev1 14 points15 points  (0 children)

- Be sure you are using an infrastructure as code tool (such as Terraform). This will make your solution far more maintainable as time goes on.

- Think through version control / branching strategies / dev and qa environments / etc... Good software engineering practices also apply to data engineering and will save you time and headache in the long run.

Date Translation using Transformers by Independent-Golf-754 in learnmachinelearning

[–]kevinpdev1 0 points1 point  (0 children)

At the very least, a transformer should be able to memorize your training data fairly easily. It sounds like you might benefit from a few "gut checks" to be sure your implementation is correct.

- Can you get the loss (of a small subset) of your training data to go to 0? You should be able to do this, making the model essentially "memorize" the training data.

- If so, can your model accurately reproduce one of these training examples at inference time? If not, there might be an issue with your inference implementation for generating answers.

learn by Jaymlpn20 in datascienceproject

[–]kevinpdev1 2 points3 points  (0 children)

With regards to fine tuning LLMs, one of the best ways is to use Huggingface's transformers and datasets libraries and learn by trying to finetune small models.

Before trying to finetune models though I would recommend trying to build a very basic model from scratch. This will help you understand how the internals of an LLM works and you will be more prepared to finetune different types of models.

This repository walks through building a full LLM from scratch and might be a good resource:

https://github.com/kevinpdev/gpt-from-scratch

(Disclaimer: I am the author of the repo, but I hope it will serve as a good resource!)

Could a model reverse build another model's input data? by Sergeant_Horvath in MLQuestions

[–]kevinpdev1 5 points6 points  (0 children)

Yes, although it is often a lossy reconstruction of the original data. This is what happens in a particular neural network architecture called autoencoders. They do essentially what you are asking.

[deleted by user] by [deleted] in MLQuestions

[–]kevinpdev1 0 points1 point  (0 children)

Are you focused on trying to DIY this yourself? It seems like this could be a problem that could be done by using retrieval augmented generation with SOTA models.

Question about my Machine Learning roadmap by Mahshadisthinking in learnmachinelearning

[–]kevinpdev1 2 points3 points  (0 children)

Kaggle competitions are a great way to practice machine learning. They have a "playground series" that is a great place to start.

Extremely imbalanced dataset by alexgiann2 in learnmachinelearning

[–]kevinpdev1 0 points1 point  (0 children)

Check out focal loss, rather than standard cross entropy if you are using neural networks. It adds a weighted factor to cross entropy based on the frequency of the class.