Derivation of the Backprop Learning Rule

David S. Touretzky 15-486/782: Artificial Neural Networks

1. Unit Activation Equation

net_{k} = ∑_{j} y_{j} · w_{jk}  (1)

y_{k} = f(net_{k})  (2)

The transfer function f(·) can be any smooth, differentiable, nonlinear function. Originally the logistic function (1+exp(−x))^{−1} was used, but many today favor tanh because its range is [−1, +1] instead of [0, 1], which gives better learning behavior.

2. Error Measure

Error E is summed over all patterns and all output units. The summation over patterns is left implicit below. d_{k} is the desired output value for unit k on the present pattern, and y_{k} is the actual output produced by unit k.

E = \frac{1}{2} ∑_{k} (d_{k} − y_{k})^{2}  (3)

3. Error of the Output Layer

δ_{k} is the gradient of the error with respect to unit k’s input. It is backpropagated to the preceding layer to calculate δ_{j}, and also used to calculate the weight update Δw_{jk}.

\frac{∂E}{∂y_{k}} = (y_{k} − d_{k})  (4)

δ_{k} = \frac{∂E}{∂net_{k}}  (5)

= \frac{∂E}{∂y_{k}} · \frac{∂y_{k}}{∂net_{k}}  (6)

= (y_{k} − d_{k}) · f′(net_{k})  (7)

4. Backpropagated Error for Hidden Units

We back-propagate the error through the w_{jk} connections to calculate the error signal for hidden unit j.

\frac{∂E}{∂y_{j}} = ∑_{k} (\frac{∂E}{∂net_{k}} · \frac{∂net_{k}}{∂y_{j}} )  (8)
= ∑_{k} (δ_{k} · w_{jk})  (9)
δ_{j} = \frac{∂E}{∂net_{j}}  (10)
= \frac{∂E}{∂y_{j}} · \frac{∂y_{j}}{∂net_{j}}  (11)
= \frac{∂E}{∂y_{j}} · f′(net_{j})  (12)

5. Weight Update

We update the weights by the negative of the error gradient (because we want error to decrease), scaled by a learning rate η.

\frac{∂E}{∂w_{jk}} = \frac{∂E}{∂net_{k}} · \frac{∂net_{k}}{∂w_{jk}}  (13)
= δ_{k} · y_{j}  (14)
\frac{∂E}{∂w_{ij}} = \frac{∂E}{∂net_{j}} · \frac{∂net_{j}}{∂w_{ij}}  (15)
= δ_{j} · y_{i}  (16)
Δw_{jk} = −η · \frac{∂E}{∂w_{jk}}  (17)
Δw_{ij} = −η · \frac{∂E}{∂w_{ij}}  (18)