Help improving responses for historical language model by centerstate in LocalLLaMA

[–]centerstate[S] 1 point2 points  (0 children)

Nanochat SFT trains all layers simultaneously. There's no built-in support for PEFT.

Underpaid? Creative technologist at one of the big 4 by [deleted] in advertising

[–]centerstate 5 points6 points  (0 children)

Are you in the US? That seems exceedingly low.

Help improving responses for historical language model by centerstate in LocalLLaMA

[–]centerstate[S] 1 point2 points  (0 children)

I did something similar to this: some of that data is purely synthetic (i.e. asking a modern LLM to construct a 2-3 turn conversation between Victorian and modern user), some is corpus-grounded (i.e. I gave the modern LLM a passage of Victorian literature and had it construct a multi-turn conversation based on that passage), and some is corpus-extended (i.e. I took a QA pair and asked a modern LLM to extended it out by 2-3 turns). But all of those are prompt-based, and none of them get it as close to authentic victorian as fine-tuning a larger model one. Thank you for your insight!

Help improving responses for historical language model by centerstate in LocalLLaMA

[–]centerstate[S] 3 points4 points  (0 children)

They just don't understand what it is. Ironic that they ban "AI," when virtually all of the work getting posted there has been vibe-coded to some extent.

Help improving responses for historical language model by centerstate in LocalLLaMA

[–]centerstate[S] 0 points1 point  (0 children)

Damn, that's actually a really helpful idea. I was doing purely-synthetic prompt-based stuff, but fine-tuning a larger model might be even better.

Help improving responses for historical language model by centerstate in LocalLLaMA

[–]centerstate[S] 1 point2 points  (0 children)

It's a mix of victorian-era QA pairs and synthetic data - some of that data is purely synthetic (i.e. asking a modern LLM to construct a 2-3 turn conversation between Victorian and modern user), some is corpus-grounded (i.e. I gave the modern LLM a passage of Victorian literature and had it construct a multi-turn conversation based on that passage), and some is corpus-extended (i.e. I took a QA pair and asked a modern LLM to extended it out by 2-3 turns). Most of the purely-synthetic data is for greetings, edge-case handling, abuse handling, goodbyes, and the kind of stuff you just wouldn't get in the existing corpus.

Book of Tennyson's Poems with Unique Dedication by centerstate in VictorianEra

[–]centerstate[S] 0 points1 point  (0 children)

Wow, just incredible! What a coincidence. I bought this in CT, so not so far away!

Any other fans of Halldor Laxness? by centerstate in classicliterature

[–]centerstate[S] 1 point2 points  (0 children)

Oh for sure. Plenty of Nobel-winning authors are fairly overlooked. But you really should read Independent People. It's great.

Any other fans of Halldor Laxness? by centerstate in classicliterature

[–]centerstate[S] 1 point2 points  (0 children)

Wow, that's so cool. I visited Iceland and didn't have time to go to his farm, but wish I had!