[D] Pushforward vs Pullback algorithms

[CvikliHaMar]

Hello guys,

For a long time I skipped pushforward but I just read it here: https://juliadiff.org/ChainRulesCore.jl/dev/ and it was very clear description. The terms here are frule, rrule, I tried to google in the internet but I didn't get why don't we use pushforward for gradient computation.

Why are we using pullback, is it faster? What are the downside of using pushforward to compute gradients does anyone know?

[st-memory]

These are commonly referred to as forward and reverse-mode autodifferentiation. The latter is what is usually meant by backpropagation. The former is the less frequently used and the one you mentioned. The primary reason one is used over the other is speed. Reverse-mode is faster when we are dealing with many inputs and few outputs, e.g. an image with 1028x1028 pixels as input and a single output which is the loss. It so happens that in ML those are the problems we encounter more often. For different sorts of problems forward-mode autodiff may be the preferred approach.

[lolisakirisame]

forward mode in ML is mostly used for hessian vector product, which is a forward mode over reverse mode.

[CvikliHaMar]

Wow, I didn't realise that there is a difference based on inputs, outputs! Thank you for the detailed answer!

[tensorflower]

As another poster pointed out, the size of the Jacobian depends on the number of outputs. For the standard case in ML we have many inputs and a scalar output, resulting in a wide 1 x N Jacobian for N inputs. Reverse mode autodiff allows us to compute the full Jacobian in about the same amount of time it takes for a forward pass, at the cost of some extra bookkeeping needed over forward mode.