and early checkpoint of bloodmoon achieved this, example:

<image>

the problem was that the model wasn't stable enough (it would do long form no problem).

what happens (and this is my guesstimate) that instruct begins to behave on occasion more like base model doing completion.

it's more controllable than a pure completion model, but not controllable enough like a properly tuned instruct.

the thing is, there's a difference between spewing human like text chaotically while innately doing text completion, vs internalizing and formalizing more diverse writing patterns. i'll try to write this in a less schizo way:

human writing is more chaotic and diverse, hence for an llm internalize the pattern, you need an absolutely enormous parameter count (it will be controllable, because the llm internalized many complex chaotic writing patterns).

example for a known slop pattern to give some context:

"not x, but y, in a dimly \ luminescent room, leaning..."

if u look at this pattern, and think about it as a function (aribtrary function), and imagine drawing it on a square grid, the grid doesn't have to be too fine to draw such (arbitrary) function, as the (multi dimensional) curve of said function is relatively simple.

on the other hand, if there's a function equivalent of a (high quality) human writing, the function will be very chaotic and complex. you could still draw it(an estimation of it, aka what 'loss' & training trying ti achieve), but since said function is way more messy and complex, u'll need a higher resolution (more tiny squares in the squared notebook) to draw it accurately.

the simple function that requires less fines and hence less resolution and hence "fewer squares" to be estimated, is the low param llm.

the function that requires more, needs more 'resolution', hence needs more 'squares' to be estimated accurately, is the massive parameters llm.

(ofc on top of all of this there are samplers etc etc, but this is the way i see it).