Somewhere between 1B - 5B parameters, transformer-based language models go from interesting to intelligent to insightful. Currently training a 3B model after having worked for a while with a sub-1B one (t5-3b / t5-large) -- the difference is palpable.
A good DALL-E 2 prompt, I promise:
Soft, warm-glow holographic reality: a cloud of small lines of neatly organized luminous text filling the space around him like speech bubbles, connecting alternate possibilities in words, floating around a student's head as he stands thinking with hands extended out in a busy but cozy candlelit workshop. Wide shot on Hasselblad Mark II, photographed from behind. Firefly swarm vibes.
Thinking like playing with clay; incrementally molding a form rather than micro-assembling.
I really like this framing of a "keyboard" for latent space navigation, vs. the generic "input method" term I've been using to think about this problem.
I’m wondering what a keyboard would like where text is manipulated on the dimension of ideas instead of characters. @thesephist's demo @betaworks tools-for-thought conference felt like an evolution of writing.
Are yet-unknown ideas and yet-undiscovered facts in between the known or outside of the known?
What should those even mean, conceptually?
Should we look closer, or should we look farther?
I need to get as fast at prototyping and validating deep learning models as I am at prototyping and validating web apps. This requires investment in:
- Knowing/understanding a few tools deeply and intimately
- Understanding core concepts deeply
- Investing in custom tooling and infrastructure where it makes sense
Generative AI technologies like GPT and DALL-E could allow a "creator economy for worldbuilding" to exist.
getting so good at life you get by without a job, you're just in demand and people want you around unblocking them, performing miracles and shit
The foundations of modern physics were laid rapidly, by a small group of brilliant people collaborating loosely and autonomously, in a few decades in the early 20th century. It bothers me that we haven't seen our understanding of nature advance so quickly after WWII. So I've been thinking about what the necessary conditions may be to create such a rapid, creative environment for discovery and progress. Here is my current hypothesis:
- A small group of the best people
- working in close collaboration and communicating frequently, but pursuing their hypotheses independently
- on a few, well-known important problems onto which their entire field is focused.
I hypothesize that similar conditions were true for computing in the late 60's/early 70's and similar conditions may be true now for ML. Loose collaboration between independent leaders across a focused field.
How might we create and nurture such conditions?