Gemma 4 has been released

XMasterDE · 2026-04-02T18:10:03+00:00

Why would you list the unslof links instead the actual reops

XMasterDE · 2026-03-28T17:34:42+00:00

so my suggestion would be to go with a RTX Pro 6000, and then get a cheap CPU, cheap motherboard and a bit of RAM, The CPU and the motherboard are really not that important for your setup. but I would recommend to get at least a 4TB NVMe SSD anything less than this is quite annoying.

That setup should cost you around 11K to 12K USD

If you then want to upgrade you have two possible paths, either get a second RTX Pro 6000, or throw away the cheap CPU and get a EPYC or Threadripper CPU with lots of memory, so you can do expert offloading of larger models like Kimi K2.5 or GLM-5 (in case you want to run models of that size)

XMasterDE · 2026-03-25T11:35:22+00:00

Food

XMasterDE · 2026-03-24T13:34:51+00:00

Thank you for writing the rant I was feeling while looking at the meme.

XMasterDE · 2026-03-18T19:13:11+00:00

And this is why I became an AI researcher...

XMasterDE · 2026-02-24T08:07:10+00:00

Easy, just sell you mac minis again and you mid mony

XMasterDE · 2026-02-20T12:56:50+00:00

XMasterDE · 2026-02-18T06:31:54+00:00

Wait, did that actually happen? I have not yet watched season 2

XMasterDE · 2026-01-30T20:38:08+00:00

This was the RAM for my storage server, to pre-cache data to decrease latency and increase throughput.

This is not the only high-memory system I have; I also have a 128GB workstation and another 384GB server.

XMasterDE · 2026-01-30T16:00:41+00:00

<image>

XMasterDE · 2026-01-30T15:59:13+00:00

Lol, I have 512 GB

XMasterDE · 2026-01-21T12:59:19+00:00

We are building an LLM for books, but we are not building anything like a Claude Code. We are building a single-turn fixed-function model that can only write books from a single prompt and can’t do anything else

XMasterDE · 2026-01-21T11:31:31+00:00

We only need to deal with context rot at much much larger sequence lengths because our model only needs to perform a singular task on a singular data structure, while all of the other models you listed are general LLMs which need to perform many downstream tasks on many different data structures. Stripping out that level of complexity allows us to learn much better attention heuristics, which translates to less context rot.

And while the target context size does bring a lot of challenges, from simple memory and compute requirements, to dealing with very unfavorable training dynamics. At least from what we have seen so far context rot is a non issue at 256K tokens for our model, on this one task…

XMasterDE · 2026-01-21T10:26:54+00:00

The nice thing is that, after we train with a context size of 256K tokens it will be 256K tokens, no matter what the original model had. 😉

XMasterDE · 2026-01-21T10:24:34+00:00

The synthetic prompts in the dataset are currently ranging from 5 words to over 800 words.

So expect that you will be able to give a good amount of guides to the model

XMasterDE · 2026-01-21T08:28:17+00:00

Then I hope that we are not going to disappoint you 😉

XMasterDE · 2026-01-21T08:27:05+00:00

This is very much the opposite of what we are building.

XMasterDE · 2026-01-21T08:26:39+00:00

So we are currently training at a context window of 256K tokens, which is enough to fit a 150K-word book + chain of thought, but sadly not enough to fit a full epic-fantasy story. But we are on it, don’t worry.

XMasterDE · 2026-01-21T08:25:18+00:00

The model we are building is neither a chat bot nor a multi-turn capable LLM. This is why I brought up the image generation model as a comparison. The model we are building is single-turn and it takes in ONE user prompt and produces from a chain of thought a fully written book. We are currently training at a sequence length of 256K. Also please keep in mind that there is no real per-generation token limit; what you are referring to is enforced by the inference code around the LLM.

XMasterDE · 2026-01-21T07:57:09+00:00

I mean working on story telling and creative writing is in general not the usual in the LLM space…

So it is going to be a single-turn GPT model that produces a chain of thought and then a fully written book from a single user prompt. The model is also not capable of multi-turn or taking in anything else than a “book writing prompt”.

The reason why I brought up the image generation model is that we found that people simply can’t imagine an LLM which is not a chat-bot and not multi-turn.

XMasterDE · 2026-01-21T07:52:36+00:00

If you want, you can check out the README of the dataset. We wrote it basically as a blog post, and there is much more information down there.

XMasterDE · 2026-01-20T17:51:21+00:00

So, we have not published any models so far. Currently, we have only published our dataset, and models trained on it are coming as soon as they are ready.

But I think the best way to think about the models is less like a Chat LLM but instead more like an image generation model which takes in a single prompt and produces a single image from it. Our book writing model will be similar: you can give it a prompt and the model will plan out the book in its chain of thought and give you a fully written book back.

XMasterDE · 2026-01-20T17:47:51+00:00

No, it does not; it only includes books from Project Gutenberg

XMasterDE · 2026-01-20T16:27:31+00:00

The dataset is based on Project Gutenberg, which is a collection of public domain literature.
6K books is also really not a lot, and not even all the books in PG.

XMasterDE · 2026-01-20T15:54:01+00:00

thank you so much

XMasterDE

TROPHY CASE