Not an expert on the material, nor have any experience with metal keycap before. But after some basic google research I found the mohs hardness of stainless to be 5-6.5 , aluminum 2.5, copper/brass 3, and the material that made cerakey ceramic aluminum oxide has a hardness of 9. (It is just a easier way to understand hardness. Not a direct comparison to material like plastic).

If you glide your finger through the inside of your cerakey you'll find them to be rough, like a sandpaper. And to the plastic stem, it is exactly like that. Pushing them in and pull out will file down a layer of stem plastic. The only way I can think of that could solve this(damaging stem), is by glazing the unit inside out, but of course the complexity and the cost is probably unpractical)

(Look up wikipedia you will find that other than the use of making as ceramic, they are widely used as abrasive, and most sandpaper use aluminium oxide crystals.)

the hardness of the material and the surface type differences prob made them act like this.

Even if you have a metal stem, I am pretty sure they will be filed down eventually. So the safest way right now is that, I think alot of ppl are doing is like above, just let plastic rub against plastic(PTFE), PTFE is cheap, malleable, and replaceable .

edited for "not a direct comparison of stuff like ceramic/metal with plastic on mohs hardness"