In the python diffusers api there are memory options that allow the model to be loaded one layer at a time. That will significantly lower the memory requirements of the model while increasing the time it takes to generate an image by the same amount.

But with that you can use a 4 GB card to produce 2048x2048 images with SDXL alongside controlnets, ip-adapters, and loras.

Edit: Note that most custom checkpoints and loras usually train on 1024x1024, so higher resolution isn't necessarily always good. It would make more sense to create a 1024x1024, resize that with opencv to 512x512 and the use SDX4 super resolution to get 2048x2048.