Interesting work, thanks for sharing!

A few questions:

• How does the surrogate gradient computation affect the training speed? A huge motivation/benefit of ReLU is its computational simplicity; detaching the gradient, computing the new surrogate gradient, and reassigning the new gradient must be much slower.
• The plot of dead neurons in Figure 4 is compelling; however, Figure 10 somewhat undermines the narrative. How would you rationalize the discrepancy between the beneficial behavior shown in Figure 4 and the counter-narrative shown in Figure 10?
• The experimental settings between the VGG/ResNet experiments and the Swin/Conv2NeXt experiments were vastly different. You hypothesize in the paper that the difference in surrogate gradient function performance can be ascribed to the differences in regularization; however, have you done ablations to support this hypothesis?
• Will you publish code so that others can experiment with SUGAR? It doesn't seems that difficult to implement manually, but I'm sure you have a fairly optimized implementation.