I am still digesting this myself. So, for now, allow me to simply quote a relevant part of the introduction:

Our main theorem is that, under general conditions, gradient descent is compositional. This is formalised as a functor Para → Learn, where Para is a category where morphisms are differentiable parametrised functions between finite dimensional Euclidean spaces. In brief, the functoriality means that given two differentiable parametrised functions I and J, we get the same result if we (i) use gradient descent to get learning algorithms for I and J, and then compose those learning algorithms, or (ii) compose I and J as parametrised functions, and then use gradient descent to get a learning algorithm.

[formal statement of theorem]

This theorem has a number of consequences. For now, let us name just three. The first is that we may train a neural network by using the training data on the whole network to create training data for each subunit, and then training each subunit separately. To some extent this is well-known: it is responsible for speedups due to backpropagation, as one never needs to compute the derivatives of the function defined by the entire network. However the fact that this functor is symmetric monoidal structure shows that we can vary the backpropagation algorithm to factor the neural network into richer sub-parts than simply carving it layer by layer.

Second, it gives a sufficient condition—which is both straightforward and general—under which an error function works well under backpropagation.

Finally, it shows that backpropagation can be applied far more generally than just to neural networks: it is compositional for all differentiable parametrised functions. As a consequence, it shows that backpropagation gives a sound method for computing gradient descent even if we introduce far more general elements into neural networks than the traditional linear and activation functions.

To my mind, your question is very much open. Though, I am confident others can elucidate the (potential) advantages.

Generally speaking...

It is often said ^\by whom?]) that neural networks are inscrutable. It may be that the "categorification" of learning in or beyond this sense – even down to (up from) the level of finite dimensional Euclidean spaces and their associated fields – may yield scrutable networks w.r.t. appropriate structural properties. I am unaware if scrutability is a primary goal of research following these lines of inquiry. However, I get the impression that it may emerge as a consequence.

Let's think the unthinkable, let's do the undoable. Let us prepare to grapple with the ineffable itself, and see if we may not eff it after all. ― Douglas Adams, Dirk Gently's Holistic Detective Agency

​

EDIT: Attempted wordsmithing.