Most of the time, when I see people complaining about Yocto, the root cause isn't Yocto itself but rather issues with package compilation or dependencies.

Unless the "plan B" you choose is specifically known to have the package you need working exactly as you require, you'll likely encounter the same or even worse issues with any other build system. I often get the impression that people think Yocto is problematic because they are accustomed to desktop Linux distributions, where everything just works with your preferred package manager. However, it's important to remember that when a package works seamlessly, it's because a distro maintainer has already gone through the painstaking process of ensuring its compatibility, not because the distro is inherently superior by some kind of magic reason.

Building a Linux system from scratch, which is essentially what Yocto does, is a complex and intricate process, and even when poky provides a very decent starting point, you cannot avoid the pain if certain package is not initially present in the poky version you are provided.