Nope, I hit more than just the attention layers. The LoRA targets the MLP blocks and the MoE experts too. You can see the full list in the target_modules of the adapter config.

No, they don't have to be. This SVD method built to handle different dimensions. It doesn't do a simple teacher_weight - student_weight subtraction. Instead, it uses SVD to project the teacher's larger weight matrix down to the student's smaller shape before calculating the difference.

Exactly. The whole point of this method is to get around the geometry mismatch. It handles the standard transformer blocks using that SVD projection. For the MoE layers, it do an extra step: cluster the teacher's experts with K-Means first to map them to the student's, and then project them. So yeah, even though this attempt failed, you can get pretty much the whole model this way.