Even when we extrapolate arbitrarily far into the future of very large, very capable ML models, as long as they are built on the current paradigm (Internet-scale self-supervision followed by fine-tuning for desired behavior), I think being Extremely Fucking Good at anything will still remain valuable, regardless of field.
At the upper echelon of any craft you're a tastemaker more than a skilled hand, and taste is like entropy — there is only so much you can distill from a limited dataset into a model before it starts losing fidelity.
Human preference data will always be a scarce resource.
I want a version control/edit history browsing interface that lets you view your file as a 2D image extruded out into 3D space into the time dimension. Instead of seeing spans of text crossed out and added, you just see them appear lower down the "time stack" and disappear higher up. And then I can imagine scrubbing through slices of this time stack with a dial or a scroll wheel like scrubbing through slices of a CT scan.
I like this because it gives physical materiality to answers to abstract questions like "how often does this file change"? or "Which of these changes were more persistent"? or "Are there more additive or subtractive edits to this project in the last month?"
People who seem like they had good ideas earlier than everyone else didn't usually have uniquely good ideas. They had ideas that seemed merely credible at the time, but had extraordinary conviction in them.
— AV.
We have to become humble in front of this overwhelming misery and overwhelming fornication, overwhelming growth, and overwhelming lack of order. Even the stars up here in the sky look like a mess. There is no harmony in the universe. We have to get acquainted to this idea that there is no harmony as we have conceived it.
But when I say this, I am full of admiration for the jungle. It is not that I hate it, I love it, I love it very much, but I love it against my better judgment.
— Werner Herzog
At risk of over-anthropomorphizing LMs, I think context window = field of view and retrieval into context = memory. You can try to have ever more expansive fields of view, but you don't really need it after a certain point if you just have good memory.