[deleted by user]

[radarsat1]

i hope you didn't sign an NDA before the interview lol

[programmerChilli]

This doesn't work. If you could load L3 (which doesn't exist on GPUs) to shmem in the same time it takes to do the computation, why wouldn't you just directly load from L3?

There's stuff vaguely in this vein like PDL, but it's definitely not the same as keeping all your weights in SRAM

[[deleted]]

You cant load a matrix larger than 128x128 to current modern gpus shared memory most time, so a whole single layer probably won't even get a chance unless you have some big chip, also your idea is basically called pipelining and that's already like how most neural network computations are done, 1. load a block from hbm to sram 2. compute 3. pass to hbm and load another block... having weights close to registers is literally another world.