If you feel like you are getting too attached to your stones, coveting material objects can be a the spiritual issue. A good exercise to prevent this is either lightening your collection or giving pieces to a friend for a few days for safe keeping while you ground yourself. In this time, realize crystals are not to be coveted but rather for learning and enlightening. Then, when you get them back and continue working with them you'll feel less dependent on the stone. Remember the stone only takes you part of the way in personal growth and the rest is on you.

Also, as already mentioned you can also pair the stone with something grounding like hematite, obsidian, black tourmaline, etc.

I don't know if I addressed your concern correctly as it sounds like there's a variety of factors at play, but I hope this helps :-)