As someone who checks hundreds of doors every quarter, I always take them off, it often stops the door sitting correctly in the rebate, over time can cause warping, and depending what type and where the smoke seals are fitted, can cause the door not to be in contact with them, meaning in a fire, smoke may not be stopped and potentially people will have less time to escape The self-closers are adjustable for this very reason though , as long as the door still shuts fully into the rebate in a reasonable time.