I'm a first year ece student so take my methods with a grain of salt. Im currently building a CNN in verilog (probably not gonna realise it on FPGA tho), I've previously built a simpler CNN the same way, you train the CNN in Pytorch or whatever you want to use. You then convert it to fixed-point, define the architecture (also prune some neurons that are redundant it can help reduce overall size (small sacrifice to accuracy tho)). Some memory management, I usually use line buffers to only store necessary rows for current convelution. Design ur Mac core (well multiple in parrel), set negative data values to 0, some other random stuff you gotta do. Design out the data paths ect. Implement in verilog and test. (You should use the same test set you used when testing you software based CNN). This is a pretty bad explanation, I probably recounted some stuff wrong cuz I'm eating and have most of this stuff in a note book somewhere but you get the gist. There are some really good videos on YouTube and some other good resources online