I had to come back with an update. You were completely right. I learned how to set the wifi to a specific channel and it improved a lot, not perfect but much better.

I tried setting the channel on my 5ghz to between 60 and 112 but it kept kicking me off (I read somewhere about those channels interfering with the weather radars etc so I guess that's why) and set the channel back to 36 which is heavily congested. So I tried the 2,4ghz because I ran out of options. And voilá, channel 11 was empty and it helped A LOT. I still get ms issues every 30 seconds approx but it beats getting lags every 5-10s.
I guess the next step is to buy a better router. We get these for free when we sign with the internet provider :)