How to do automatic differentiation?
This is not a package, just an idea how such a package might work. You could define a custom value type autoDiffValue
that stores a value and it's first derivative (with respect to some variable), then define arithmetic operations for that type:
Clear[autoDiffValue]
autoDiffValue /: autoDiffValue[a_, adx_] + autoDiffValue[b_, bdx_] :=
autoDiffValue[a + b, adx + bdx]
autoDiffValue /: autoDiffValue[a_, adx_] * autoDiffValue[b_, bdx_] :=
autoDiffValue[a*b, adx*b + a*bdx]
autoDiffValue /: autoDiffValue[a_, adx_] + b_ :=
autoDiffValue[a + b, adx]
autoDiffValue /: autoDiffValue[a_, adx_]*b_ :=
autoDiffValue[a*b, adx*b]
autoDiffValue /: autoDiffValue[a_, adx_]^b_ :=
autoDiffValue[a^b, a^(b - 1)*adx*b]
Using that and your function h
, you can then call:
h[{1, 2, 3}, autoDiffValue[5, 1]]
and get
autoDiffValue[86, 32]
i.e. the value and the first derivative of h[{1, 2, 3}, x]
at x=5
You could automate generating derivative definitions:
Clear[defineFunc]
defineFunc[fn_] :=
autoDiffValue /: fn[autoDiffValue[a_, adx_]] =
autoDiffValue[fn[a], D[fn[a + x*adx], x] /. x -> 0]
defineFunc /@ {Sin, Cos, Exp};
(it's probably possible to extend defineFunc
so it can create definitions for Plus
, Times
, ... too)
This is more or less the same technique you would use to define AD in e.g. C++ using operator overloading. It should work in principle, although a lot of work would still be needed to implement multivariable and higher derivatives and to support vector and matrix arguments (so calculations can be done with packed arrays).