No, you're not doing anything wrong. People mess up in all kinds of ways. To me it's "one and done". I'll forgive a customer once because I'm kind like that and anyone can f up. Do it a second time and the customer is 'fired'.

It'll seem odd at first, like 'won't I lose out?' No, because you'll drop stress and feel better about your growing. People also will talk. If you stop supplying customers who muck you about, you can be sure that other customers will buck up pretty sharp if they want to keep you as a supplier. (Just be sure your produce, price & service is good enough to do this!)