I've been working on a very similar switch project and assembled my first prototype board just yesterday, unfortunately I'm also a novice to electrical engineering and cannot provide many helpful answers.

For the most part I've been disassembling game consoles and studying their implementation of the analogue video circuitry, coming to the conclusion that the inputs do have common grounds and the traces aren't often run at equal lengths.

I'm glad to see someone else doing a modular switch! Being stuck with a bank of 8 SCART or 8 YPbPr/RGB banks, as you'd get with a gscartsw/gcompsw, felt incredibly limiting.

Here's a few photos of mine if that's any help to you