The structure of neural network error surfaces

Analysis of the error surfaces of feed-forward neural networks is complicated by the high dimensionality of the weight space. Visualisation over one- and two-dimensional slices, and Monte Carlo analysis of stationary points can produce misleading results. We show that, in some situations, important features of the error surface can only be visualised by considering the error over non-planar manifolds of weight space. We also show that Monte Carlo simulations can depend critically upon the random step size chosen. The relationship can reveal key properties of the local structure of the error surface.