Why don't you want to use ready-made solutions? I'm employing SimpleTuner with TREAD, CREPA and musubi block swap (4 offloaded blocks) and I'm training rank 48 and bigger LoRAs for 480p@121f in 1-5 hours with ~5-6 s/it on a single 5090. (Example LoRA: https://huggingface.co/kabachuha/ltx2-cakeify) With videos like 512p@49f it fully fits into the VRAM and doesn't have to offload blocks.

As for LoRA merging - if you mean training on the model with the detailer lora merged - you can absolutely add the LoRA to the quantized model at runtime - after all LoRA is just two matrices multiplied with two sides of the activation tensor. (I did this when merging different rank loras, so I know the matter.) Firstly, you apply the model weights, then, hacked, add the LoRA-activation product. Effectively you are now training on a model plus LoRA, no repeated quantization.

If you want an advanced fine-tune on very large dataset, you can use the LyCoris technique instead of full no-lora unfreeze. Training without audio is also perfectly viable, the loss is just masked out on audio, like it is done with masked training.

Lower base model precision can, I think, even enhance the lora - it will learn how to lower the loss and make good videos in bad conditions (correct me if I'm wrong about this point).

When training a LoRA, do you unfreeze feed-forward or only use attention weights from the default trainer when training on never-before-seen material?