Visualizing the
Loss Landscape
of Neural Nets
The Problem
A neural network with millions of parameters exists in an unimaginably high-dimensional space.
Training a neural network means finding a set of weights that minimizes the loss function— a measure of how wrong the network's predictions are. But how does gradient descent navigate this vast, complex terrain?
We can't visualize millions of dimensions directly. But we canslice through this space and see what the landscape looks like around a trained network's solution.
Why do some networks train easily while others get stuck? The answer lies in the shape of their loss landscape.
The Method
We create a 2D slice through the high-dimensional loss landscape using two random direction vectors.
By sampling at many points on this 2D plane, we build a surface that reveals the local geometry of the loss landscape around our trained solution.
Filter-wise Normalization
A crucial insight: random directions must be normalized to match the scale of each filter in the network.
Without normalization, large filters dominate the visualization. With filter-wise normalization, each filter contributes proportionally to its learned magnitude.
Without Normalization
Random noise overpowers the structure. The landscape appears artificially rough.
With Normalization
The true geometry emerges. We can now see meaningful differences between architectures.
For each filter :
The Reveal
Watch the landscape transform as we toggle skip connections.
ResNet-56
Chaotic — riddled with local minima and sharp ridges. Optimization is treacherous.
ResNet-56
Smooth — a gentle bowl with clear gradients pointing toward the minimum.
This is why ResNets train so much better than plain networks. Skip connections fundamentally reshape the optimization landscape.
Optimizer Trajectory
Watch SGD navigate the loss landscape in real-time.
The glowing path traces the optimizer's journey from a random starting point toward the minimum. On a smooth landscape, the path is direct. On a chaotic one, it wanders.
The trajectory is projected onto the same 2D plane as the loss surface, showing how the optimizer's path appears when viewed through our visualization lens.
Citation & Resources
BibTeX
@inproceedings{li2018visualizing,
title={Visualizing the Loss Landscape of Neural Nets},
author={Li, Hao and Xu, Zheng and Taylor, Gavin and Studer, Christoph and Goldstein, Tom},
booktitle={Advances in Neural Information Processing Systems},
pages={6389--6399},
year={2018}
}