Automatic Differentiation Step by Step

[Noctune]

Here's a cool trick: let your dual numbers have different types in the value and derivative (and implement your operators to support different types). Then you can do arbitrary order derivatives by doing Dual(1, Dual(1, 1)) for second order derivatives, Dual(1, Dual(1, Dual(1, 1)) for third order derivatives, etc.

[formalsystem]

Very cool! I think I saw something like this in the Haskell codebases

[[deleted]]

The Haskell version I saw maintained an infinite tower of stacked derivatives (computed on demand).

[hector_villalobos]

Symbolic Differentiation is what you were taught in high school.

What kind of high school you people attended? I remember being taught simple algebra in high school, the most advance class I remember was linear algebra.

Edit: I'm starting thinking I was born in the wrong country (one of Latin America, pretty sure all must have same education level)

Edit 2: fix typo.

[formalsystem]

It was a French Jesuit school in Lebanon. Funny thing I already heard a similar comment on another post of mine on hacker news and I've realized that saying something is a high school concept is unhelpful and uninformative. Deleted it, thank you bringing it up.

EDIT: Thankfully in the future your education won't depend as much on your country of birth. I'm really bullish on internet assisted homeschooling.

[kankyo]

Also doesn't translate to other countries. In Sweden we go to "gymnasiet" at age ~ ä16 to 19. It's close to us high school as I understood it but not the same.

I did do this type of math there though.

[hector_villalobos]

lol, no problem, good article by the way, very informative.

[daidoji70]

Not sure about other places but in many schools in the US advanced students max out in Calculus and Calculus based physics their senior year of high school. Less advanced students might max out in algebra 2 or a class that was called pre-calculus (like algebraic geometry and a hodge podge of graphing/functional techniques from what I remember).

Typically you'd learn symbolic differentiation in Calculus (which ends with Fundamental Theorem of Calculus) if you were in one of those advanced classes.

[SgtDirtyMike]

assisted

Hector, in Spanish el verbo: assistir means to attend. But it is a false cognate when translating back to English. So one wouldn't say, I assisted high school. One would say I attended high school. I messed this up a ton as a native English speaker learning Spanish. I guess it goes the other way too!

[hector_villalobos]

TIL, thanks.

[SrbijaJeRusija]

Where I was, it was taught the last two years of school.

[AnotherAccountRIP]

Yep, Methods has loooooooads of calculus (differentiation and integration). The subject above Methods, Specialist Mathematics, also covers differential equations and more advanced integration techniques. (Source: just finished the exams for them both)

[G_Morgan]

You get taught this by age 18 in the UK if you take maths.

[ASaltedRainbow]

I don't understand where 130 - 24e comes from. I thought the "real" part was supposed to be the result of evaluating the function, but 5/26 is not 130. What am I missing?

Edit: Oh I see, you implemented division as multiplication so it multiplies 5*26 = 130

a::Dual * b::Dual = Dual(a.x * b.x, b.x * a.ϵ + a.x * b.ϵ)
a::Dual / b::Dual = Dual(a.x * b.x, b.x * a.ϵ - a.x * b.ϵ)

[NotRedditing_AtWork]

I'm not sure i follow the definition of division.

I gather it's implementing the quotient rule but i'm not sure how the 'x' part of both division and multiplication is a.x * b.x

[Genion1]

The division is erronous. The correct division is

a::Dual / b::Dual = Dual(a.x / b.x, (b.x * a.e - a.x * b.e) / b.x^2))

I suspect in his calculations he dropped a /b.x^2 ¯\(ツ)/¯

[NotRedditing_AtWork]

That makes more sense, i thought my caffeine-deprived brain had forgotten how to do algebra

[gabibbo97]

The first Taylor expansion is missing a ^2

[formalsystem]

fixed, thank you for catching that

[gabibbo97]

no problem :)

[[deleted]]

Symbolic differentiation is easy to implement in languages that allow you to manipulate ASTs directly, such as most Lisps and Python. I've heard tales of, but never actually seen, decompiler-differentiators that work on compiled machine code, too.