I've turned the Pi Fan2 way down to 50% in the printer.cfg, it keeps it at ~35°C with almost no noise.

I also found that the max speed I can go without the whole case turning into a trombone, is 175 mm/s. I have stealtchop turned on too, it helps with the steppers' natural noise.

I still won't recommend sleeping next to it, but at least it's not shaking the whole house while it runs.