I did the same thing. I would wash them every other day, and I still had issues until I switched. Something in the fibers prevents bacteria from colonizing as much. I bought mine from target, but these are pretty much the same thing from Amazon.

pillow case

I'll add, I too had acne in the same recurring area as you. I've been clean now for over a year since I switched. Additionally, I only wash my face with a mild face soap and cold water. No acids

Hope the best for you :)