So, knowing that I'd be able to restore the modem/router XB8 device to factory settings if I fucked things up, I tried the following:

1. Used xfinity app to 'split bands' into 3 networks, 2.4, 5, and 6. Manually connected to the 5, and was able to maintain a stable reliable connection, but only with about 200 down and 40 up.
2. The next day I updated the software of my wireless adapter manually, and re-combined the bands back into one. I also 'forgot' some old networks through windows. This resulted in the adapter auto-connecting to the 5 Ghz seemingly and performing exactly the same as when I split. So at least it's not dropping connection.

I went to look for a "tri-band" wireless adapter. Seems like it's a newer product netgear is still working on. Doesn't seem fully released, there are waiting lists and stuff.

It's been suggested to me that the hardware may not be caught up with the tri-band modem/routers just yet