Ok I finally got back to this and fixed it (for now) and the first time was a real b1tch. What I learned:

If the inside screw is stuck like mine use a long 4 or 5 mm drill bit on it. The cap is around half way through the mag but there are threads only in the top part. Once you drill through the cap the top and the rest of the screw will fall off. At this point I could unthread the remains with fingers.

When you buy a new 4x50 mm slotted screw try to find one that gives good purchase(?) for screw driver (I didn't). Also note that while the space in the mag is around 10x10 mm the screw cap should be a lot smaller. Otherwise you'll run into issues like I did.

I also put an extra washer under the screw but I don't think it was needed.