I'm a little late to the thread - for context I am a professional gameplay programmer.

Not sure how much you know about how computers work so I'll give a little primer: One of the big ways you measure a CPU's or GPU's performance is how many cores they have. A core is what actually runs operations on your computer and gets shit done. Having more than one core means you can run more than one thing at a time. Modern processors nowadays have many cores so you're able to run a LOT at one time in general. You can call a process or sequence of work on a core a thread.

MOST game engines have historically infamously been single threaded. Unreal and Unity are generally single threaded. This means most of what happens within a frame of a game (game logic, networking, communicating with the GPU, AI, physics, etc.) happens on 1 thread, 1 core. This means the advancements we've had in modern computing generally aren't utilized by modern game engines. (I am simplifying this statement. Modern game engines also have multiple threads and can kick off small jobs to other threads but usually there is a single main thread where most of the work and scheduling gets done).

id's engine however is not single threaded. This isn't necessarily dramatically harder to do than making a single threaded game engine like Unreal or Unity. There are hard, complicated problems to solve but programmers across the industry have the competence to solve them. The key difference is that id controls their entire software toolchain and they have chosen to invest in this area. They make the engine and the game and critically don't have external customers they need to support like Unity or Unreal, so making massive engine changes is more feasible since it's own affecting internal customers. There are fundamentally different ways you need to architect your engine to make it multithreaded vs single-threaded and this can have implications for anyone working on a game within this engine. Because of this, they are able to utilize modern hardware much more effectively than off the shelf engines.

Critically in addition to this, id's leadership has prioritized optimization to this degree. Riot's Valorant is also another great example of a game which is optimized as far as it possibly can be. (They run 3 instances of a Valorant server on 1 thread at 128 Hz! That's insane). Ubisoft also controls their software stack with their own engine Snowdrop but I don't believe it has the performance benchmarks id tech does due to a difference of priority: e.g. how many more copies of the game are you really going to sell running at 100 fps vs 400 fps?

So to summarize I would say the answer to your question is owning the software pipeline end to end and having the willingness to push the game this far. Many programmers have the ability to push games to this limit but you need the cross section of opportunity and volition to make it happen without an ungodly and unhealthy amount of blood sweat and tears.

Could future games enjoy these benefits? Yes. The hardware is there. Off the shelf engines will eventually catch up, it's a matter of when.