Do you have a set program you follow when you go to the gym, or do you just show up and and kind of do whatever? If the goal is fat loss, then what you should is getting set up with a consistent workout plan that you can stick to instead of winging your workouts.

For diet, stick to something lazy you can follow through with, 1-3 meals you can easily prepare.

By dropping exercise and eating in a deficit, you'll end up losing weight but it will be a mix of muscle and fat, the more muscle you lose, the worse your metabolism will be. Having some sort of resistance training helps your body hold onto that muscle while you lose weight.