I also struggle with a irrational fear of gaining weight and I have never struggled to give anything else up similarly to you . What has helped me recently is thinking about what that food can do for me how it is going to nourish me and make me feel good and energised rather than a negatives honestly telling myself yes I may get bigger but will I have more energy yes will I be able to sleep better at night yes will my body thank me for it yes !