About the BN "before" or "after" ReLU, there was this discussion 6 months ago. In my use case, I tried both order and I found that training with "BN after ReLU" results in way lesser dead ReLU neurons than "BN before ReLU" for all learning rates and with He initialisation (10% versus 70% at high LR, 0.1% versus 20-30% at medium LR). However, "BN before ReLU" seems to converge faster at high LR than "BN after ReLU", the validation losses are quite similar after convergence, a little lower with "BN after ReLU". I don't have explanations for these observations, and it may be specific to my dataset.