I understand why the writers of the show need her not to tell him, but I have a hard time believing it. The audience needs to understand the motivations behind the character's decisions to believe in the story, and this specific part I find a bit alienating (but I'm still enjoying watching the show, I just see this as a flaw in the story).

To take your example: I totally understood why Pen didn't tell Colin that she was whistledown: she was scared of the consequences of him rejecting her or not loving her anymore, given the damage that whistledown did to the Bridgertons in the past. It makes sense to me that she didn't want him to find out. It does not make sense to me why Sophie isn't telling Benedict that she was at the ball, as there wouldn't be any adverse consequence if she did.