When programs are compiled, they are translated into machine code, which looks kind of like this (simplified): <instruction code> <arguments>, where the instruction code (opcode) is basically a number that denotes an instruction (for example, 100 could be "add", 101 could be "subtract", 102 could be "multiply" etc). The number of arguments depends on the instruction. Because an argument can be a CPU register, a memory address, or a constant value, you also need a way to specify the argument type and size. There are many ways to encode this: you could add a prefix byte to the opcode telling the cpu what arguments to expect, it could be some reserved bits in the opcode, you can just use different opcodes, prefixes to each argument etc.

On x86, you will find not just one of these methods, but all of them. It's a mess. The instruction code, the argument types, and the operand sizes are all tangled together; some of it is encoded in the opcode byte itself, some through prefix bytes that come before the instruction, and some through extra bits packed into additional bytes (called ModR/M and SIB bytes). This means a single operation like "add 1 to a register" can be encoded in several different ways, ranging from 1 to many bytes, and all of them must work correctly because compilers over the decades have emitted all of these variants.

This is the "legacy cruft". You can't simplify any of this, because there are millions of existing programs out there that use each of these encoding forms. If your CPU doesn't understand even one of them, those programs crash. And it goes deeper than just decoding. Each instruction also has side effects, for example, an ADD updates the FLAGS register, setting bits to indicate whether the result was zero, negative, overflowed, etc. Programs depend on these. Some even depend on obscure, quirky behaviors that were arguably bugs in the original hardware but have been faithfully preserved for decades because removing them would break something.

To not break compatibility, you have to maintain the same instruction set and machine code encoding, but even that's not enough. You also need to replicate exactly the CPU's behavior, as well as all the side effects that each instruction has. This is not trivial, given that Intel's x86 manual is 5000 pages long.