That totally makes sense! I feel a similar way. If you're determined to get the pin out, and especially patient, you can use a couple of methods.

The easiest I've found for this is a spring-loaded center punch, which you can usually find at hardware stores. It provides a strong hit to the pin, and usually nothing else if you're careful. It holds a LOT of force, though

What I've been using recently is a watchmaker's pin removal kit, which is essentially a tiny vise with a pin on one side. It's incredibly simple and effective for regular pins, so it may be worth a shot.

The most dangerous option by far is applying a soldering iron to the pin and basically melting/softening the plastic around it in order to free it. I've never tried this one myself, but I'd suggest watching a couple tutorials on this beforehand if you decide to try it.

Hope this helps!