2 Derivation of backproagation equations in fully-connected neural networks
2.1 Abstract
In this chapter, a detailed derivation of the backpropagation equations is presented for the fully-connected dense neural networks. It is derived from scratch with the assumption of knowledge in the mathematical form of the fully-connected neural networks. The derivation was first done in element-wise form, then later transformed to the vector form. The algorithms for the loss gradient computations for both the forms were given in the end.
Here, both the fully-connected neural networks and the dense neural networks represent the same type of network. Hence, their names will be used interchangeably in the text.
2.2 Deriving of backpropagation equations
Consider a dense neural network having \(L\) layers, with \(N^{(l)}\) neurons on each layer, \(1 \le l \le L\). The notations \(\mathbf{W}^{(l)}\) and \(\mathbf{b}^{(l)}\) represent the weights matrix and bias vector of the \(l\)th layer. And the index notations \(i\) represents the previous \(l-1\)th layer, \(j\) represents the current and \(k\) represents the next layer.
Now, the weight matrix of an \(l\)th layer is defined as
\[ \begin{aligned} \mathbf{W}^{(l)} = \begin{bmatrix} w_{1,1}^{(l)} & \dots & w_{1,N^{(l-1}}^{(l)} \\ \vdots & w_{j,i}^{(l)} & \vdots \\ w_{N^{(l)},1}^{(l)} & \dots & w_{N^{(l)},N^{(l-1)}}^{(l)} \end{bmatrix} \end{aligned} \tag{2.1}\]
Similarly, the bias vector of an \(l\)th layer is defined as
\[ \begin{aligned} \mathbf{b}^{(l)} = \begin{bmatrix} b_1^{(l)} \\ \vdots \\ b_j^{(l)} \\ b_{N^{(l)}}^{(l)} \end{bmatrix} \end{aligned} \tag{2.2}\]
Now, the loss function for the \(m\)th data point is defined as \(L_m = L_m(y_m,f_{NN}(x_m))\) assuming \(x\) and \(y\) are the inputs and outputs of the network. Then the cost function is defined as the sum of loss functions over all the data points.
\[ J(\mathbf{W},\mathbf{b}) = \sum_{m=1}^N L_m \tag{2.3}\]
The aim is to find the weights \(\mathbf{W}\) and biases \(\mathbf{b}\) for the entire network that minimizes Eq. 2.3 i.e. optimal weights
\[ \mathbf{W}^*,\mathbf{b}^* = \underset{\mathbf{W},\mathbf{b}}{\arg\min} \ J(\mathbf{W},\mathbf{b}) \]
and
\[ \nabla_{\mathbf{W},\mathbf{b}} J(\mathbf{W},\mathbf{b}) = \sum_{m=1}^N \nabla_{\mathbf{W},\mathbf{b}} L_m . \]
Taking one element from Eq. 2.1 as \(w_{j,i}^{(l)}\),
\[ w_{j,i}^{(l)} := w_{j,i}^{(l)} - \alpha \frac{\partial J}{\partial w_{j,i}^{(l)}} \tag{2.4}\]
here, \(\alpha\) is the learning rate, \(i=1,2,\dots,N^{(l-1)}\), \(j=1,2,\dots,N^{(l)}\) and for all layers \(l=1,2,\dots,L\).
Rewriting Eq. 2.4 with \(J\) expanded as
\[ w_{j,i}^{(l)} := w_{j,i}^{(l)} - \alpha \sum_{m=1}^N\frac{\partial L_m}{\partial w_{j,i}^{(l)}} \tag{2.5}\]
Now, to find \(\partial L_m/\partial w_{j,i}^{(l)}\), the chain rule has to be applied to all the functions that lie between \(L_m\) and \(w_{j,i}^{(l)}\). Here, the \(w_{j,i}^{(l)}\) dependence chain is given as
\[ w_{j,i}^{(l)} \to z_j^{(l)} \to \mathbf{z}^{(l)} \to \mathbf{a}^{(l)} \to \dots \to L_m \]
So, the chain rule from backwards is given as
\[ \frac{\partial L_m}{\partial w_{j,i}^{(l)}} = \frac{\partial L_m}{\partial \mathbf{a}^{(L)}} \frac{\partial \mathbf{a}^{(L)}}{\partial \mathbf{z}^{(L)}} \frac{\partial \mathbf{z}^{(L)}}{\partial \mathbf{a}^{(L-1)}} \dots \frac{\partial \mathbf{z}^{(l)}}{\partial z_{j,i}^{(l)}} \frac{\partial z_{j,i}^{(l)}}{\partial w_{j,i}^{(l)}} \]
which can be written in a compressed form as
\[ \frac{\partial L_m}{\partial w_{j,i}^{(l)}} = \frac{\partial L_m}{\partial \mathbf{z}^{(l)}} \frac{\partial \mathbf{z}^{(l)}}{\partial w_{j,i}^{(l)}} \tag{2.6}\]
The derivation of Eq. 2.6 is given below.
First, \(w_{j,i}^{(l)}\) is used by \(z_j^{(l)}\) so,
\[ \frac{\partial L_m}{\partial w_{j,i}^{(l)}} = \frac{\partial L_m}{\partial z_j^{(l)}} \frac{\partial z_j^{(l)}}{\partial w_{j,i}^{(l)}} = \frac{\partial L_m}{\partial z_j^{(l)}} a_i^{(l-1)} \ . \tag{2.7}\]
Let,
\[ \frac{\partial L_m}{\partial z_{j}^{(l)}} = \delta_j^{(l)} \]
Now, \(z_j^{(l)}\) is first used by \(a_j^{(l)}\) as \(a_j^{(l)} = h(z_j^{(l)})\), where \(h(.)\) is the activation function. So,
\[ \delta_j^{(l)} = \frac{\partial L_m}{\partial a_j^{(l)}} \frac{\partial a_j^{(l)}}{\partial z_j^{(l)}} = \frac{\partial L_m}{\partial a_j^{(l)}} h'(z_j^{(l)} \tag{2.8}\]
Now, \(a_j^{(l)}\) is used by \(z_1^{(l+1)},z_2^{(l+1)},\dots,z_{N^{(l+1)}}^{(l+1)}\). So, to find \(\partial L_m/\partial a_j^{(l)}\)
\[ \begin{aligned} \frac{\partial L_m}{\partial a_j^{(l)}} &= \frac{\partial L_m}{\partial z_1^{(l+1)}} \frac{\partial z_1^{(l+1)}}{a_j^{(l)}} +\frac{\partial L_m}{\partial z_2^{(l+1)}} \frac{\partial z_2^{(l+1)}}{a_j^{(l)}} + \dots + \frac{\partial L_m}{\partial z_{N^{(l+1)}}^{(l+1)}} \frac{\partial z_{N^{(l+1)}}^{(l+1)}}{a_j^{(l)}} \\ &= \delta_1^{(l+1)} \frac{\partial z_1^{(l+1)}}{a_j^{(l)}} + \delta_2^{(l+1)} \frac{\partial z_2^{(l+1)}}{a_j^{(l)}} + \dots + \delta_{N^{(l+1)}}^{(l+1)} \frac{\partial z_{N^{(l+1)}}^{(l+1)}}{a_j^{(l)}} \\ & = \sum_{k=1}^{(N^{(l+1)}} \delta_k^{(l+1)} w_{k,j}^{(l+1)} \end{aligned} \]
So, writing Eq. 2.8 as
\[ \delta_j^{(l)} = \left[\sum_{k=1}^{N^{(l+1)}} \delta_k^{(l+1)} w_{k,j}^{(l+1)}\right]h'(z_j^{(l)}) \tag{2.9}\]
For the bias term,
\[ \begin{aligned} \frac{\partial L_m}{\partial b_j^{(l)}} &= \frac{\partial L_m}{\partial z_j^{(l)}} \frac{\partial z_j^{(l)}}{\partial b_j^{(l))}} \\ &= \delta_j^{(l)} . 1\\ &= \delta_j^{(l)} \end{aligned} \tag{2.10}\]
And for the last layer \(L\)
\[ \begin{aligned} \delta_j^{(L)} &= \frac{\partial L_m}{\partial z_j^{(L)}} \\ &= \frac{\partial L_m}{\partial a_j^{(L)}} \frac{\partial a_j^{(L)}}{\partial z_j^{(L)}} \end{aligned} \]
The above gradients are explicit to compute as they depend on the loss function and the final layer activation functions respectively.
2.2.1 Pseudocode for the scalar form
Forward propagation
Back propagation
2.2.2 Backpropagation equations in vector form
Writing the Eq. 2.5 in vector form for all the elements
\[ \mathbf{W}^{(l)} := \mathbf{W}^{(l)} - \alpha \sum_{m=1}^{N} \nabla_{\mathbf{W}^{(l)}} L_m \tag{2.11}\]
To find \(\nabla_{\mathbf{W}^{(l)}} L_m\)
\[ \begin{aligned} \nabla_{\mathbf{W}^{(l)}} L_m &= \begin{bmatrix} \frac{\partial L_m}{\partial w_{1,1}^{(l)}} & \dots & \frac{\partial L_m}{\partial w_{1,N^{(l-1)}}^{(l)}} \\ \vdots & \ddots & \vdots \\ \frac{\partial L_m}{\partial w_{N^{(l)},1}^{(l)}} & \dots & \frac{\partial L_m}{\partial w_{N^{(l)},N^{(l-1)}}^{(l)}} \\ \end{bmatrix} \\ &= \begin{bmatrix} \delta_1^{(l)} a_1^{(l-1)} & \dots & \delta_1^{(l)} a_{N^{(l-1)}}^{(l-1)} \\ \vdots & \ddots & \vdots \\ \delta_{N^{(l)}}^{(l)} a_1^{(l-1)} & \dots & \delta_{N^{(l)}}^{(l)} a_{N^{(l-1)}}^{(l-1)} \\ \end{bmatrix} \\ &= \mathbf{\delta}^{(l)} \left(\mathbf{a}^{(l-1)}\right)^T \end{aligned} \tag{2.12}\]
The weights gradient is obtained using outer product as shown in the last step above.
To find \(\delta^{(l)}\) vector, from Eq. 2.9
\[ \begin{aligned} \delta^{(l)} = \begin{bmatrix} \delta_1^{(l)} \\ \vdots \\ \delta_{N^{(l)}}^{(l)} \end{bmatrix} = \begin{bmatrix} \left(\sum_{k=1}^{N^{(l+1)}} \delta_k^{(l+1)} w_{k,1}^{(l+1)}\right) h'(z_1^{(l)}) \\ \vdots \\ \left(\sum_{k=1}^{N^{(l+1)}} \delta_k^{(l+1)} w_{k,N^{(l)}}^{(l+1)}\right) h'(z_{N^{(l)}}^{(l)}) \\ \end{bmatrix} \end{aligned} \]
Using Hadamard product (element-wise product) the above expression can be written as
\[ \delta^{(l)} = \left(\mathbf{W}^{(l-1)}\right)^T \delta^{(l+1)} \odot h'(\mathbf{z}^{(l)}) \tag{2.13}\]
And bias vector will be
\[ \nabla_{\mathbf{b}^{(l)}} L_m = \delta^{(l)} \tag{2.14}\]
2.2.3 Pseudocode for the vector form
Forward propagation
Back propagation
2.3 Conclusion
This chapter has the indept derivation of backpropagation equations for the fully-connected neural networks. This derivation was learnt from Dr. Sumitra’s course on Graphical Models and Deep Learning, and was digitized to preserve the notes. The next step will be to use these gradients for the weight updates using different algorithms, that will be covered in the next chapter.
