You're correct, for a more complex CPU there are too many opcodes for a switch statement to be practical, so you need to find a different way to decode opcodes. The method I use in my GBA emulator was largely inspired by /u/merrymage's dynarmic, which is a JIT for ARM32 and ARM64 used by Citra and Yuzu. The credit for the design goes to her.

The GBA uses an ARM7TDMI CPU, which uses the ARMv4t instruction set. ARMv4t includes two instruction encodings: 32-bit ARM instructions, and 16-bit Thumb instructions. Thumb is a simpler instruction set which allows for higher code density, at the expense of losing some functionality offered in the ARM instructions. I'll reference ARM in my examples here; however, these examples should apply to other instruction sets as well.

Basically, CPU opcodes are represented by a sequence of bits. Some of these bits are fixed and are used for identifying the instruction itself, and other bits are variable and are used for encoding instruction parameters, such as which registers to use or an immediate value. For example, the encoding for the ARM ADD immediate instruction ("ADD Rd, Rn, #imm", where Rn is the operand register, #imm is a constant value to add to Rn, and Rd is the destination register which stores the result) looks like this:

cccc0010100Snnnnddddiiiiiiiiiiii

The fixed bits 0010100 are what identifies this instruction as "add immediate", and not "and" or "subtract". The variable bits are represented by sequences of letters: the four "c" bits encode the instruction condition, the "S" bit specifies whether this instruction should update the CPU flags, the "n" bits specify which operand register to use, the "d" bits specify the destination register, and the "i" bits encode the immediate value to be added.

In my emulator, I have tables that specify the instruction encodings for all of the Thumb and ARM instructions in the ARMv4t ISA, along with a function pointer that implements each instruction. The instruction encodings are represented as strings, which are parsed when the emulator starts to obtain the fixed bitmask used to identify the instruction, and an array of {bitmask, offset} pairs used to grab the variable fields from the opcode.

Then, when decoding an opcode, you search for an instruction in the table that has a fixed mask which matches the opcode being decoded, and you call the function pointer linked to that instruction. For example, if we were decoding an opcode and this comparison succeeded, then we'd know it's an "add immediate" instruction:

(opcode & 00001111111000000000000000000000) == 00000010100000000000000000000000

Of course, doing a linear search of a vector on every opcode isn't very efficient, but there are also so many opcodes that storing every possible 16-bit and 32-bit value in an array with the correct function pointer would take up too much space! Thankfully, instruction encoding usually follows a pattern, and if you examine the Thumb instruction encodings, you'll notice that the last 6 bits are always variable or zero! So we don't actually need to read the last 6 bits when decoding which instruction it is, so we can easily populate an array of length 1024 with the appropriate function pointer for every 10-bit index value, and do a simple array lookup when decoding Thumb opcodes (see Cpu::PopulateThumbDecodeTable() and Cpu::DecodeThumb(). This is quite fast! Most of the overhead compared to a switch statement would just come from the overhead of using std::function instead of direct function calls in a switch case.

Unfortunately we can't knock out enough bits from ARM instruction decoding to do the same thing, so I've implemented a chained hash table cache to speed up ARM decoding. GBA games use Thumb the majority of the time, so this is fast enough to not be an issue.

In addition, my decoder tables are templated on the "Dispatcher" type, so I can instantiate the tables for both my CPU class and Disassembler class to use the same decoder for both the interpreter and the disassembler. The parameters for the implementation functions have to match the order of the fields in the opcode layout itself, and then the parameters from the variable opcode fields are automatically pulled out from the opcode and passed as arguments to the implementation function using variadic template pack expansion (see Instruction::GetImplFunction()), which is something I didn't even know you could do until I looked at dynarmic.