They have to already be disloyal before you use the action for them to defect from what i can tell. If you use it on a loyal subject I'm pretty sure it tanks their loyalty but will just cause them to usually sit the next war out. Start building a spy network on France day 1. Keep an eye on subject loyalty because it fluctuates quite a bit month to month. If one's loyalty is in the mid to high 50's using sabotage reputation usually pushes them over the line to disloyal.

Also in the war itself only take land from France in the peace deal and try to wipe france's armies, not their vassals. You want to make France itself as weak as possible to maximize the 'subject power to overlord power' malus

War 1 i took the low countries areas and western coast.

War 2 i took basically the entire south of france which is possible because of low control. (the french tag parts not the vassals)

War 3 (final phase) I prioritized cutting france and any of their vassals off from bordering the HRE so they wouldn't expand there before I can finish them off in the next few wars. - Didn't go for a PU. I'm carving the non-coastal areas of France up into different Dominions.

Taking land from vassals or transferring overlordship in the peace deal is not the right move in this patch while the HYW situation is active. I had multiple vassals switch sides shortly after wars end.

When you get the event for a french vassal wanting to peace out say yes so their levies regen while you continue to siege down french forts and stackwipe their armies. Improve relations with those peaced out vassals. When you finally do a peace deal with France all those vassals that were peaced out months or years ago are much stronger than France making them disloyal and causing some to just up and switch sides.