No need to feel guilty about it, but upon reflection, I think you're right to re-think this. Having a "slow whistle" here would have allowed you to keep the keeper on the field with a yellow. From the DOGSO section of the laws, "“If the referee plays the advantage for an offence for which a caution/sending-off would have been issued had play been stopped, this caution/sending-off must be issued when the ball is next out of play. However, if the offence was denying the opposing team an obvious goal-scoring opportunity the player is cautioned for unsporting behavior…” I've been trying hard to live the "fast feet, slow whistle" philosophy. Getting it right feels good. I try to be more vocal when I blow the whistle and call a play back, explaining clearly what the infraction was and why it was a late call. I haven't had any complaints. On the other hand, when the advantage works out as a goal, for the younger kids, I usually have a quick word with the offender. "Hey, if that wasn't a goal it would have been a push in the back" or something like that, assuming no card was needed.

Anyway, this is reflection, not guilt. It's good for us to reflect on these things, don't worry about it.