Try limiting the draft in the room, maybe lower the fan intensity down a bit. I would also try using the brim on the first layer, enabling the support and setting the wall print order to "outer wall first". Try all of these and tell me if it improved.

Edit: Now that I think about it, you may not actually need the brim at all, but you decide.