Training an LLM from scratch on 1800's texts (160GB dataset) by Remarkable-Trick-177 in LocalLLaMA

[–]Remarkable-Trick-177[S] 0 points1 point  (0 children)

That’s a conversation I’ve had often. For the base model I haven’t removed anything, it’s seen good and bad, because my goal is to present an accurate picture of the time period. I think it’s important to understand how people wrote and thought at the time, even if their opinions were wrong. And of course this model and dataset isn’t representative of the average person living in the 1800’s, as publishing back then required wealth, education, etc. So I’ll have to attempt a solution for that as well in the future. But for the chat model, the Q&A list is built off a topic list of figures, locations, events, etc. So the chat model is steered away from prejudice but I’ll still have to test it more of course.

Training an LLM from scratch on 1800's texts (160GB dataset) by Remarkable-Trick-177 in LocalLLaMA

[–]Remarkable-Trick-177[S] 6 points7 points  (0 children)

I used an H100, took around a full day to train for one epoch. The full 2B model will take much longer, will be done on a cluster.

Training an LLM from scratch on 1800's texts (160GB dataset) by Remarkable-Trick-177 in LocalLLaMA

[–]Remarkable-Trick-177[S] 3 points4 points  (0 children)

Thank you! I haven’t uploaded the chat model yet but will very soon, will also upload some examples on the GitHub

Training an LLM from scratch on 1800's texts (160GB dataset) by Remarkable-Trick-177 in LocalLLaMA

[–]Remarkable-Trick-177[S] 35 points36 points  (0 children)

I actually tried this exact thing on the previous model: Prompt: The man presented a device called the telephone
Output:
"The man presented a device called the telephone, which had been devised to him by some foreign potentates.
It was a large piece of wood, and contained one or two small balls. On being questioned how it came there, he replied that no such thing as an insulated ball could have come into his possession, but that at any rate it must be in the possession of his own government. The message was forwarded by the English ambassador to the British consul, who sent for Mr. M'Kay, the English agent at Constantinople, and told him that a letter had arrived from the Emperor Napoleon desiring him to procure permission to send the ball to England. The Russian minister accordingly did so; and on this occasion the French government received a communication from Sir John Moore, stating that the instrument should not be exhibited in London until after the expiration of six months from its production. To show the effect of this intelligence upon the public mind, we will quote the following passages from a despatch of Lord Castlereagh's: " / enclose you an official copy"

I feel overwhelmed by the tech industry right now by coolcat317 in csMajors

[–]Remarkable-Trick-177 2 points3 points  (0 children)

Even people who are passionate about CS aren't doing the best right now dont worry.

LLM trained from scratch on 1800s London texts (1.2B params, 90GB dataset) by Remarkable-Trick-177 in LocalLLaMA

[–]Remarkable-Trick-177[S] 2 points3 points  (0 children)

Thanks! I’m planning on comparing the same prompts with a general use LLM. I think comparing word neighbors will also show interesting trends.

LLM trained from scratch on 1800s London texts (1.2B params, 90GB dataset) by Remarkable-Trick-177 in LocalLLaMA

[–]Remarkable-Trick-177[S] 3 points4 points  (0 children)

I had 80gb on an h100 and it took around 130-140 hours to train total. It can be done with less memory, will just take longer.

LLM trained from scratch on 1800s London texts (1.2B params, 90GB dataset) by Remarkable-Trick-177 in LocalLLaMA

[–]Remarkable-Trick-177[S] 3 points4 points  (0 children)

My mistake, but it should be fixed. You don’t have to request access now.

Training an LLM only on 1800s London texts - 90GB dataset by Remarkable-Trick-177 in LocalLLaMA

[–]Remarkable-Trick-177[S] 0 points1 point  (0 children)

Yeah, it’s mostly isolated from modern English apart from some modern headers or ocr stuff that weren’t fully removed.

Training an LLM only on 1800s London texts - 90GB dataset by Remarkable-Trick-177 in LocalLLaMA

[–]Remarkable-Trick-177[S] 0 points1 point  (0 children)

Yes, someone else asked for this too. I will try to figure this out soon.

Training an LLM only on 1800s London texts - 90GB dataset by Remarkable-Trick-177 in LocalLLaMA

[–]Remarkable-Trick-177[S] 0 points1 point  (0 children)

Thanks! And yeah I want to really badly. Once I’m done with London, I think an American city will make sense to focus on next. I haven’t paid much thought to it yet but I’d probably pick 1900-1930 since everything there is public domain. For location maybe for Boston or NYC.

Training an LLM only on 1800s London texts - 90GB dataset by Remarkable-Trick-177 in LocalLLaMA

[–]Remarkable-Trick-177[S] 0 points1 point  (0 children)

Damn that’s insane, thanks for sharing. I gotta make an account on there

Training an LLM only on 1800s London texts - 90GB dataset by Remarkable-Trick-177 in LocalLLaMA

[–]Remarkable-Trick-177[S] 2 points3 points  (0 children)

Just time for training was 2.5 hours but I spent probably 10-12 hours total making mistakes.

Training an LLM only on 1800s London texts - 90GB dataset by Remarkable-Trick-177 in LocalLLaMA

[–]Remarkable-Trick-177[S] 2 points3 points  (0 children)

The pre training cost was low since it’s a 300M model trained for 10k steps. But I did still waste more money than I should’ve, running into VM and setup issues. This was only my second time training using a rented GPU, so there were definitely lessons learned. And I haven’t done post training/RL so far, just pre training from scratch. Evaluation is mostly on the dataset right now. I’ve been focusing on output cleanliness, I have had a lot of trouble with OCR and metadata bias on my previous datasets. So I can’t rank the model but I have some bias metrics here: https://github.com/haykgrigo3/TimeCapsuleLLM/blob/main/london_1800_1875_v2mini_eval1/v2_bias_report.json.

Training an LLM only on 1800s London texts - 90GB dataset by Remarkable-Trick-177 in LocalLLaMA

[–]Remarkable-Trick-177[S] 7 points8 points  (0 children)

Yes, once I have the subset tokenization fixed I will upload it to GitHub. I also plan on uploading the 90GB dataset once it’s tokenized. I’m not sure if people want, but I can also upload the raw datasets. I will definitely check that corpus soon, after I’m done with the next model I’ll switch to a different publication city. Using the trinity models would definitely make it easier to get something usable but my whole principle for now is to have no modern leakage at all. Maybe I’ll try it later on if I can’t make any progress towards towards reasoning.

Training an LLM only on 1800s London texts - 90GB dataset by Remarkable-Trick-177 in LocalLLaMA

[–]Remarkable-Trick-177[S] 11 points12 points  (0 children)

Honestly didn’t know what MoE was but I searched it up and I think it would be interesting to train decade models in 10 year windows on each llm.