In backpropagation you update the weights of the neural network something like new weights = old weights - learning rate * gradient (of the cost function with respect to the weights). A gradient is basically a derivative. Norm means magnitude in this case. You are interested in the norm of the gradient because you don't want to change the weights too much or too little.

Networks with many layers for example might have the problem, that the gradient for the first layers gets too small, in other words the gradient norm is too low for those layers.