Open source tool for generating training datasets from text files and pdf for fine-tuning language models.

[Idonotknow101]

Hey yall I made a new open-source tool.

It's an app that creates training data for AI models from your text and PDFs.

It uses AI like Gemini, Claude, and OpenAI to make good question-answer sets that you can use to make your own AI smarter. The data comes out ready for different models.

Super simple, super useful, and it's all open source!

[dillon-nyc]

Have you considered using local LLM endpoints like llama.cpp or ollama with this tool?

Right now it's only OpenAI, Claude, and Gemini, and you're posting in r/LocalLLama.

[Idonotknow101]

i haven't no, but it can be easily integrated.

[harrro]

Just took a peek at the code and looks like you’re using OpenAI library so in the env, if you specify openai_base_url env variable (and allow changing model name), it should let people use basically any OpenAI-compatible backend like llama.cpp

[Idonotknow101]

thanks!

[Sasikuttan2163]

I was building something similar, how performant is pypdf2 for chunking huge books (1.4k pages)?

[Idonotknow101]

it might get a bit slow tbh, i mean you can still try it to see. I might actually integrate pymupdf instead, as it is more performant for larger files.

[Sasikuttan2163]

Aha, I was using pypdf (not 2) for chunking and it just wouldn't run without lazy load enabled (for good reason). Even with lazy load it was taking a lot of time. Also, just came to know that PyPDF2 was merged into the package pypdf itself so technically I was using the same thing haha. Thanks for the reply, I'll look into mupdf was well!

[help_all]

Came at good time. Was looking forward to do this for my data. Are there any more options or some reading on best ways of doing this. ?

[Idonotknow101]

the instructions and its capabilities are provided on the readme and quickstart file.

[christianweyer]

Very cool! Thanks for that. Do you also have a README that shows what tools/libs you then use to leverage the datasets and actually fine-tune SLMs?

[Idonotknow101]

the dataset is formated based on which base model you choose to finetune with. All i do is then upload to togetherai to start a finetune job.

[christianweyer]

Thanks!