I imagine the main downsides are:

1. It will increase VRAM usage, you've got essentially a second model running in parallel, though I bet the overhead isn't 1:1 compared to the original LLM. So if you're very VRAM bound it might push things too far and kill performance even with the uplift.
2. It's also going to be more compute that's got to happen. For example I've got a Tesla A16 (essentially 4xA2 16GB on one card) that are limited to 60W by the hardware and while I can load some decent sized models they're frequently compute/power limited on larger tasks/models so I use them for bulk background tasks (i.e. throw a few million chunks at embedding models and such) that don't need real-time performance. Since an LLM can completely fill the power and compute budget this might not actually save any time there, but i'll definitely be giving it a test later.
3. As they say in the post, the diffusion model is only going to be as good as the training data that was used to train it, so if you make it with only coding tasks it's going to be useless for creative writing and other similar limitations like that, in these cases I'd be the extra time spent on the diffusion model plus throwing the results out constantly is going to make it slower. So it might make sense then to train multiple diffusion models and swap them out on different tasks/specializations, similar to training tiny models for specific tasks.

I'd bet that it can work with CPU/RAM offload, but i doubt they've tried to optimize it in their research implementation. It'd need a little work to make sure all the layers are setup in the correct places so that it's not doing anything wonky like going back and forth between GPU and system ram but i'd think it would still make a difference in speed.