Skip to content

ArianKhorasani/broken_neural_scaling_laws

This branch is 1 commit behind ethancaballero/broken_neural_scaling_laws:main.

Folders and files

NameName
Last commit message
Last commit date
Aug 3, 2023
Oct 30, 2022
Aug 22, 2023
Aug 22, 2023
Jan 11, 2023
Oct 27, 2022
Nov 2, 2022

Repository files navigation

Code Release for "Broken Neural Scaling Laws" (BNSL) paper (arxiv.org/abs/2210.14891)

Read Appendix A.6 of arXiv version of this paper for more details on how to use this code.

To reproduce the Fitting and Extrapolation of BNSL on 4 Digit Addition from Figure 5 Left, run

python fit_bnsl_and_extrapolate__4_digit_addition__dataset_size_x-axis.py

To reproduce the Fitting and Extrapolation of BNSL on a noiseless simulation of the scaling behavior of 4 Digit Addition from Figure 5 Right, run

python fit_bnsl_and_extrapolate__4_digit_addition__dataset_size_x-axis__noiseless_simulation.py

To reproduce the Decomposition of BNSL into Power Law Segments from Figure 1, run

python make_figure_1__decomposition_of_bnsl_into_power_law_segments.py

Note:

🚨🚨🚨

When you fit a BNSL to your own scaling data, you may need to adjust the grid search range and resolution to get a good fit.

🚨🚨🚨

Here is some bibtex to use for citation:

@inproceedings{
caballero2023broken,
title={Broken Neural Scaling Laws},
author={Ethan Caballero and Kshitij Gupta and Irina Rish and David Krueger},
booktitle={The Eleventh International Conference on Learning Representations },
year={2023},
url={https://arxiv.org/abs/2210.14891}
}

About

Code Release for "Broken Neural Scaling Laws" (BNSL) paper

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%