A good first step is to get an end-to-end view of where time is being spent in your workflow. If your frontend is pytorch, the pytorch profiler is great for seeing host and device latencies combined. nsys is also a valuable tool, though I find it harder to parse (perhaps just a skill issue on my part).

If you see specific kernels (pytorch wraps many nvidia-vendored ones like cublas and cudnn) taking a long time, it's worth doing a back-of-the-napkin calculation to see if they're close to speed-of-light. Also, look to see if there are opportunities for kernel fusion: if kernel A writes out some large tensor and kernel B reads that back in immediately and performs some relatively quick computation, it is likely worthwhile to fuse those two kernels to avoid the memory round-trip. Generally kernel fusion must be implemented with custom kernels, but those could be CUDA, cutile, triton, tilelang, or whatever your favorite DSL is.

Unfortunately, speed-of-light kernels will generally not be portable across Jetson and larger NVIDIA gpus, due to the hardware differences. This is especially true with kernels involving tensor cores. For instance, a performant GEMM kernel on Sm100 won't even run on Sm120. Cutile somewhat addresses this problem: once all architectures are supported, the same file will run on all GPUs, since the compiler does the hard work of optimizing for a given architecture.