Double descent

Test error falls as the model grows, spikes when it can exactly fit every training point, then falls again — double descent.

Model width w 32

Weight decay 0

test error train error

The spike is where model width equals the number of training points, w = n.