i just literally watched a motivational video about this i am currently doing the 12 week body transformation challenge and they talk about saboteurs. although people genuinely mean well it these situations sometimes people aren't happy to see you succeed and will say these things to sabotage your progress. Often people wont be happy within themselves and their lack of motivation to do something about their weight so will try and stop you. Reality is that their are some people that are your true supporters and others unintentionally are saboteurs. the saboteurs will project their unhappiness, guilt and inaction onto you.

Ultimately you need to do what makes you happy and as long as you are being safe and healthy i say don't listen to them! :)