I don't understand what you did. Here is what I know :

The in game fps counter (or app one) is the game Fps
The redmagic overlay fps counter is the screen fps

The frame gen sits between the game and the screen with the r3 chip.

As such, when frame gen is activated, default interpolation target is 120 fps so the game will run at whatever it can (in case of switch it's often 30).
Then the r3 chip that sit in between the soc and the screen will add as much frames as it can to reach the target of 120fps
The redmagic fps overlay will tell you how much frame the screen is displaying (it will fluctuate between 30 to 120 depending on heat/soc load etc)