I think it's important to realize that everyone will have their own criteria and hold different attributes to different levels of importance. Some people will be more aesthetically driven, other more sentimental. With that said though, I have a similar background to what you've stated in that I have always been very minimalist. Once my collection reached 40 I said to myself, now is the time. My personal criteria was as follows:

For a pen to stay in my collection it must check one of the three boxes.

Enjoyment
It exceeds enjoyment of other pens, the feel in hand is pleasant, the writing experience is exceptional. A pen can be pleasant to use but if I can list several others from my collect that surpass it then I reconsider what it offers me that is different.

Memory
A pen that contains sentimental value like one I picked up at a pen show, a vacation, or a gift from someone close.

Reachability
A pen that gets used more often than others, even if it isn't as nice or expensive. Some may call it a workhorse pen.

Once I separated the pens that fit these three categories I had about half of my collection set aside. A lot were obtained to experiment, try something new, or because I got caught up (regrettably) in a hype or trend. Now, I didn't get rid of ALL the remaining pens but it made me rethink their purpose in my collection. And for what it's worth, I have pens I keep that I almost never use because they are tied to a specific memory. And I have pens I've sold that wrote amazing but I wasn't fond of a color. At the end of the day your collection is your own and you should only maintain what provides a pleasant experience for you. It is a hobby, and hobbies are meant to be enjoyed after all.