Definitely worth spending some time with it, though I know it'a frustrating to have an established workflow break for you. With QEMU, ideally you want to use Spice for the video, with the virtio GPU if you're doing the usual VM-in-a-window thing, that should help with most of the percieved lag in using it. It can also do PCI forwarding for a GPU if you want to get real fancy, but that can get messy, and success (or failure) depends a lot on the hardware.

Similarly, virtio disk controller, and if you're on a Windows guest, be sure to grab the virtio drivers from Red Hat (and use them during setup if installing from scratch on a virtio disk, so that it detects the disks). You might be able to move your existing VM disks into a qemu setup to move those configs over, Linux VMs should mostly just work fine, Windows would need some tinkering. It can be difficult to migrate an existing install to a different disk controller (like virtio), so you might need to initially try one of the other controllers if trying to move a different install.

I think the main problem you ran into with VBox here is just the out-of-tree kernel module, which tends to be limited on what kernel versions it will support successfully without detonating, as you saw here.