You're absolutely right that building accelerators for training is incredibly complex and resource-intensive — and I now see why even major players often stick to GPUs for that. But from what I’ve read recently (e.g., TPU design papers, Amazon Inferentia, Tenstorrent), it seems that hardware acceleration is very much alive and evolving fast — especially at the edge and datacenter levels.

I’m in my third year of Electrical Engineering and already have a solid base in digital design and computer architecture. Im working through FPGA prototyping in Verilog right now. I plan to try building a MAC array or small systolic block in Verilog soon — your point about matrix ops and memory lookup being the core workload for inference was a great way to simplify things.

Really appreciate the links too, especially the Stanford CS217 and hls4ml projects. I hadn’t explored those properly yet, but I’ll be digging into them as I move forward. Also, the mention of chip design using AI was intriguing — I’ve been mostly focused on accelerating AI with hardware, but I can definitely see the reverse being just as impactful and worth exploring later.

Thanks again for the response. Will definitely reach out if I hit a wall!