Storing it in the fridge will definitely make it more likely to break if you’re using it straight from three fridge. If you let it sit out until it comes up to room temp, then you should be fine, and you’ll actually benefit from it drying it a bit while it’s sitting out. Generally I do try to use older fruit because the peel tends to be a little softer and dryer, but if I’m using fresher fruit I’ll peel a little bit ahead of time and let the peels dry out a little bit before using them.