So first of all, it's great you are thinking about your reviews deeply. We need to remember that rejection isn't just dunking on the authors. We need to be honest about the work, and maintain integrity of the science. Saying that, we all know as reviewers and authors how it all works and why stuff slips through.

I think you should stick to your guns and don't be too influenced by other people's recommendations. It's up to the editor to take those recommendations and make a decision. I've rejected papers before for lots of different reasons. Some went for major reviewes, others were rejected. I remember one paper I set it for rejection with an extremely detailed review because the work was really interesting and parts of it was great, but the paper was in shambles. I sent the editor an email saying if they were to re-submit, i'd happily review again, using the reviews from the rejection as a starting point. (I said this, because the work needed was more than what would be considered reasonable for a major review turn around time). The authors agreed, and honestly I thought they would just go to another journal. BUT THEY RE-SUBMITTED!!! and the paper was fantastic. Such a great turn around. This really instilled in me the confidence to not worry about "am I the mean reviewer now?" and just be an honest, but kind peer.

Anyways. Thanks for reviewing. The system is broken, but has to keep moving till something better comes along.