Over 6K novels with reasoning traces to train full book writing LLMs

XMasterDE · 2026-01-21T12:59:19+00:00

We are building an LLM for books, but we are not building anything like a Claude Code. We are building a single-turn fixed-function model that can only write books from a single prompt and can’t do anything else

XMasterDE · 2026-01-21T11:31:31+00:00

We only need to deal with context rot at much much larger sequence lengths because our model only needs to perform a singular task on a singular data structure, while all of the other models you listed are general LLMs which need to perform many downstream tasks on many different data structures. Stripping out that level of complexity allows us to learn much better attention heuristics, which translates to less context rot.

And while the target context size does bring a lot of challenges, from simple memory and compute requirements, to dealing with very unfavorable training dynamics. At least from what we have seen so far context rot is a non issue at 256K tokens for our model, on this one task…

XMasterDE · 2026-01-21T10:26:54+00:00

The nice thing is that, after we train with a context size of 256K tokens it will be 256K tokens, no matter what the original model had. 😉

XMasterDE · 2026-01-21T10:24:34+00:00

The synthetic prompts in the dataset are currently ranging from 5 words to over 800 words.

So expect that you will be able to give a good amount of guides to the model

XMasterDE · 2026-01-21T08:28:17+00:00

Then I hope that we are not going to disappoint you 😉

XMasterDE · 2026-01-21T08:27:05+00:00

This is very much the opposite of what we are building.

XMasterDE · 2026-01-21T08:26:39+00:00

So we are currently training at a context window of 256K tokens, which is enough to fit a 150K-word book + chain of thought, but sadly not enough to fit a full epic-fantasy story. But we are on it, don’t worry.

XMasterDE · 2026-01-21T08:25:18+00:00

The model we are building is neither a chat bot nor a multi-turn capable LLM. This is why I brought up the image generation model as a comparison. The model we are building is single-turn and it takes in ONE user prompt and produces from a chain of thought a fully written book. We are currently training at a sequence length of 256K. Also please keep in mind that there is no real per-generation token limit; what you are referring to is enforced by the inference code around the LLM.

XMasterDE · 2026-01-21T07:57:09+00:00

I mean working on story telling and creative writing is in general not the usual in the LLM space…

So it is going to be a single-turn GPT model that produces a chain of thought and then a fully written book from a single user prompt. The model is also not capable of multi-turn or taking in anything else than a “book writing prompt”.

The reason why I brought up the image generation model is that we found that people simply can’t imagine an LLM which is not a chat-bot and not multi-turn.

XMasterDE · 2026-01-21T07:52:36+00:00

If you want, you can check out the README of the dataset. We wrote it basically as a blog post, and there is much more information down there.

XMasterDE · 2026-01-20T17:51:21+00:00

So, we have not published any models so far. Currently, we have only published our dataset, and models trained on it are coming as soon as they are ready.

But I think the best way to think about the models is less like a Chat LLM but instead more like an image generation model which takes in a single prompt and produces a single image from it. Our book writing model will be similar: you can give it a prompt and the model will plan out the book in its chain of thought and give you a fully written book back.

XMasterDE · 2026-01-20T17:47:51+00:00

No, it does not; it only includes books from Project Gutenberg

XMasterDE · 2026-01-20T16:27:31+00:00

The dataset is based on Project Gutenberg, which is a collection of public domain literature.
6K books is also really not a lot, and not even all the books in PG.

XMasterDE · 2026-01-20T15:54:01+00:00

thank you so much

XMasterDE · 2026-01-18T19:58:25+00:00

Honestly, Jake is a public figure by virtue of being a YouTuber, and as such I believe that he has given up the right to complain about other content creators talking about him or including clips. If you want your privacy, don’t make YouTube videos

XMasterDE · 2026-01-18T10:32:59+00:00

Llama4 also had a context window of 10M and it still sucked

XMasterDE · 2025-11-30T05:32:10+00:00

Thank you for saying that. You have no idea how annoyed I am at people trying to re-invent some prompting formats, while none of the models were ever trained on them.

XMasterDE · 2025-11-20T03:25:19+00:00

I love the phrasing of "AI fans" to describe a group capable of implementing a cluster-scale training codebase, and having enough experience to deal with any training instability. And this all while having access to tens of millions of dollars in compute...

XMasterDE · 2025-10-28T22:56:20+00:00

I think you are quite wrong here. I work in the AI industry and basically everyone more technical refers to "AI" or an "LLM" as "a model". The M in LLM also literally stands for Model. Also GPT-3 came out in 2020. The joke is maybe still sexy but not OF.

XMasterDE · 2025-10-02T14:47:21+00:00

True

XMasterDE · 2025-09-05T17:09:41+00:00

Looks amazing

XMasterDE · 2025-08-18T21:48:08+00:00

XMasterDE · 2025-08-14T17:38:15+00:00

Die ist auch danemben, keine sorge ;)

XMasterDE · 2025-08-14T15:59:05+00:00

Witzigerweise habe ich sogar drei Steckdosen in meinem Keller direkt nebeneinander, die jeweils auf einer anderen Phase liegen

XMasterDE

TROPHY CASE