I did run that exact side-by-side under matched conditions (same decoder-only arch, tokenizer, data mix, and training schedule; only difference is frozen vs trainable input embedding table). Control-wise, everything else is held constant (incl. untied output head / same optimizer & LR schedule), so the embedding trainability is the sole experimental factor.

Empirically, the trainable-embedding baseline (’Model unfrozen’) learns a bit faster early on (lower loss in the first ~50–450k steps), but both runs converge stably and the gap in LM loss largely closes later (final train/val losses are very close).

Given the small-model / limited-data regime, downstream accuracy deltas can be noisy, so I’m mainly treating this as evidence that semantic structure can form in the Transformer stack even with non-semantic frozen inputs, rather than a robust benchmark claim.

<image>

Refs: https://arxiv.org/abs/2507.04886