Machine Learning 2026 · Week 06University of Bern · Interactive lab

Chapter 7 · Supervised learning

Deep learning

A neural network is a stack of matrix multiplications and elementwise nonlinearities, trained by gradient descent with gradients from backpropagation. Start from a single ReLU neuron, widen it into a two-layer network, see what activations and depth do, watch a network learn its own feature map, then step through backpropagation by hand.

7.1–7.2 · A neural network with a single neuron

One ReLU, one kink

House prices cannot be negative, so we fit \(\bar h_\theta(x)=\max(wx+b,0)\) instead of a line. Left: the fit. Right: the mean-square loss \(J(w,b)\) over all parameter pairs; drag the dot or run gradient descent. In the grey region every house has \(wx+b\le0\): the neuron is “dead”, its gradient is zero and gradient descent cannot leave.

\[\bar h_\theta(x)=\operatorname{ReLU}(wx+b)\]\[J(\theta)=\frac1n\sum_{i=1}^n\big(h_\theta(x^{(i)})-y^{(i)}\big)^2\]Eqs. (7.1), (7.2), (7.11)
Presets
What to look for

Parameters θ = (w, b)

Gradient descent (7.8)

Size is in units of 100 m², price in 100 kCHF. Drag the dot in the loss landscape to choose (w, b).

Fit and loss landscape houseReLU(wx + b)dead region
J(w, b), darker = lower

Check yourself

Discussion questions

Try each question on paper first. Then open the worked answer: it has a derivation, a live calculation you can change, and a one-line takeaway.

D1Why not use the identity as activation?Take a two-layer network with \(\sigma(z)=z\) and zero biases. What function class does it represent?Worked answer
  1. With \(\sigma(z)=z\): \(\bar h_\theta(x)=W^{[2]}a^{[1]}=W^{[2]}\sigma(z^{[1]})=W^{[2]}z^{[1]}\).
  2. Using \(z^{[1]}=W^{[1]}x\): \(\bar h_\theta(x)=W^{[2]}W^{[1]}x=\tilde Wx\) with \(\tilde W=W^{[2]}W^{[1]}\).
  3. Any number of layers collapses the same way, and with biases the result is affine: \(\tilde Wx+\tilde b\).
  4. So depth adds nothing: the network is a linear model, whatever its width. Nonlinear activations are what make depth useful.
Current result, W⁽¹⁾ = [[1, 2], [−1, 1]], W⁽²⁾ = [3, −1]
TakeawayWithout a nonlinearity, a deep network is one matrix in disguise.
D2How many parameters does a fully connected network have?Input dimension \(d=m_0\), layer widths \(m_1,\dots,m_r\) (with \(m_r\) the output width).Worked answer
  1. Layer \(k\) has \(W^{[k]}\in\mathbb R^{m_k\times m_{k-1}}\) and \(b^{[k]}\in\mathbb R^{m_k}\): \(m_km_{k-1}+m_k=m_k(m_{k-1}+1)\) numbers.
  2. Summing over layers gives \(\sum_{k=1}^r m_k(m_{k-1}+1)\).
  3. For the housing network of (7.14): \(d=4\), \(m_1=3\), \(m_2=1\) gives \(3\cdot5+1\cdot4=19\).
  4. The count grows with the product of neighbouring widths, so doubling every width roughly quadruples the parameters.
Current result, scalar output
TakeawayParameters live in the weight matrices, so the count is dominated by the widest pair of adjacent layers.
D3Derive the backward function of \(\mathrm{MM}_{W,b}(z)=Wz+b\).Given the upstream gradient \(v=\partial J/\partial u\) with \(u=Wz+b\), find \(\partial J/\partial z\), \(\partial J/\partial W\) and \(\partial J/\partial b\).Worked answer
  1. Componentwise \(u_j=\sum_i W_{ji}z_i+b_j\), so \(\partial u_j/\partial z_i=W_{ji}\), \(\partial u_j/\partial W_{ji}=z_i\) and \(\partial u_j/\partial b_j=1\).
  2. Chain rule (7.57): \(\frac{\partial J}{\partial z_i}=\sum_j W_{ji}v_j\), i.e. \(\frac{\partial J}{\partial z}=W^\top v\).
  3. Each \(W_{ji}\) affects only \(u_j\): \(\frac{\partial J}{\partial W_{ji}}=v_jz_i\), i.e. \(\frac{\partial J}{\partial W}=vz^\top\). Likewise \(\frac{\partial J}{\partial b}=v\).
  4. All three cost \(O(mn)\), the same as the forward product: this is why backprop is as cheap as the forward pass up to a constant.
Live check against finite differences, W ∈ ℝ²ˣ³, J = vᵀ(Wz + b)
TakeawayBackward through a linear layer is multiplication by the transpose, and the weight gradient is an outer product.
D4Show that the logistic loss has gradient \(g(t)-y\) with respect to the logit \(t\).\(\ell(t,y)=-y\log g(t)-(1-y)\log(1-g(t))\) with \(g(t)=1/(1+e^{-t})\).Worked answer
  1. The sigmoid satisfies \(g'(t)=g(t)(1-g(t))\).
  2. \(\frac{\partial}{\partial t}\big[-y\log g(t)\big]=-y\frac{g'(t)}{g(t)}=-y(1-g(t))\).
  3. \(\frac{\partial}{\partial t}\big[-(1-y)\log(1-g(t))\big]=(1-y)\frac{g'(t)}{1-g(t)}=(1-y)g(t)\).
  4. Adding: \(-y+yg+g-yg=g(t)-y\). This is the starting value \(\partial J/\partial z^{[r]}\) of the backward pass in Algorithm 3.
Current result
TakeawayThe error signal is simply “predicted probability minus label”: large when confidently wrong, near zero when confidently right.
D5In what sense is layer normalization scale-invariant?Ignore γ, β and consider \(\mathrm{LN}_0(z)=(z-\mu(z))/\sqrt{v(z)+\epsilon}\).Worked answer
  1. For \(\alpha>0\): \(\mu(\alpha z)=\alpha\mu(z)\) and \(v(\alpha z)=\alpha^2v(z)\).
  2. With \(\epsilon=0\) and \(v(z)>0\): \(\mathrm{LN}_0(\alpha z)=\frac{\alpha(z-\mu)}{\alpha\sqrt{v}}=\mathrm{LN}_0(z)\). Adding a constant to every entry is removed by the mean, too.
  3. With \(\epsilon>0\): \(\mathrm{LN}_0(\alpha z)=\frac{z-\mu}{\sqrt{v+\epsilon/\alpha^2}}\). The invariance is approximate and breaks when \(\alpha^2v\) is comparable to \(\epsilon\).
  4. The output has \(\frac1m\sum_i\hat z_i^2=v/(v+\epsilon)\), which equals 1 only when \(\epsilon=0\).
Current result from Experiment 6
TakeawayLayerNorm removes the scale and offset of its input; ε only matters when the input is nearly constant.
D6How many parameters does a convolutional layer save?Compare Conv1D-S on \(z\in\mathbb R^m\) with a dense \(\mathrm{MM}\) layer, then count the multi-channel layer (7.54).Worked answer
  1. Conv1D-S equals \(Qz+b\mathbf1\) with a banded Toeplitz \(Q\): only \(k\) distinct values plus one bias, so \(k+1\) parameters.
  2. A dense layer \(\mathbb R^m\to\mathbb R^m\) has \(m^2+m\) parameters and costs \(O(m^2)\); the banded product costs \(O(km)\).
  3. With \(C\) input and \(C'\) output channels there is one length-\(k\) filter per channel pair and one bias per output channel: \(kCC'+C'\).
  4. The parameter count does not depend on the signal length \(m\): the same filter is shared across all positions.
Current result
TakeawayConvolution trades a full matrix for a small shared filter: locality and parameter sharing.