Figure 11 - available via license: Creative Commons Attribution 4.0 International
Content may be subject to copyright.
The prior combining sines and lines introduced in Section 4.3 deteriorates in accuracy as more data points are conditioned upon, as detailed in Section 5.2. Notably, the neural network approximation makes an erroneous approximation here and still predicts a sloped sine. This model was trained on dataset with up to 100 examples, thus this is not due to length generalization.
Source publication
Traditionally, neural network training has been primarily viewed as an approximation of maximum likelihood estimation (MLE). This interpretation originated in a time when training for multiple epochs on small datasets was common and performance was data bound; but it falls short in the era of large-scale single-epoch trainings ushered in by large s...
Similar publications
A fundamental question in interpretability research is to what extent neural networks, particularly language models, implement reusable functions via subnetworks that can be composed to perform more complex tasks. Recent developments in mechanistic interpretability have made progress in identifying subnetworks, often referred to as circuits, which...
