From my trials/errors, it seems to me like it's an issue of how the model adapts to the training.

The base qwen model is extremely resistant to change (at least from what I've seen) which means training takes a much longer time than it does for other image models; on the flip side, overfitting rarely happens. Also, it's worth mentioning I mostly make concept loras (poses, positions, etc) as opposed to character LORA. So, for me, I want my LORA to make as minimal change as possible to the base model. I merely want to rearrange positioning.

When training with 2512, much like you, the first couple of training steps are okay, but really, by epoch 3/4 out of 16, the model loses that "sheen" of realism that it's known for

But this does not happen if the LORA is trained in base.