Updates on 2022/12/31

Ink & Switch and Anthropic produce far and away the best-written and best-produced research reports in computer science-related fields (as far as I can tell), and everyone else should strive to reach for the same level of presentation, accessibility, clarity, and depth.

Underrated fact about training in the very large regime: you don't have to worry about overfitting/early stopping because single-epoch training is the default, and it turns out it's No Big Deal at all if you do single-digit number of epochs on these huge AF overparameterized models!

Academic benchmark datasets that are in the order of tens of thousands of samples are annoying in this way.