Added a few functions for drawing histograms into the Oak standard library, and now I can do this!
std.range(100000) |>
std.map(random.normal) |>
debug.histo({ bars: 20, label: :start, cols: 50 })... which renders ...
7 ▏
38 ▏
175 ▌
574 █▋
1740 █████
3951 ███████████▍
7804 ██████████████████████▌
12066 ██████████████████████████████████▉
15913 ██████████████████████████████████████████████
17296 ██████████████████████████████████████████████████
15783 █████████████████████████████████████████████▋
11614 █████████████████████████████████▋
7160 ████████████████████▊
3562 ██████████▎
1527 ████▍
577 █▋
159 ▌
40 ▏
12 ▏
1 ▏Might our descendants look upon our probable emergence and departure from Earth's gravity well the same way we look upon our ancestors' emergence and departure from the ocean — as a necessary proving ground that we outgrew?
I was trying to develop an intuition for how "hard" masked language modeling and masked text reconstruction (autoencoding) are, so I wrote a little script to "mask" a certain % of words from stdin.
{
println: println
default: default
map: map
stdin: stdin
constantly: constantly
} := import('std')
{
split: split
trim: trim
join: join
} := import('str')
Cli := import('cli').parse()
MaskFraction := float(Cli.opts.'mask-fraction') |>
default(float(Cli.opts.m)) |>
default(0)
stdin() |>
trim() |>
split('\n') |>
map(fn(l) l |> split(' ')) |>
map(fn(l) l |> map(fn(w) if rand() < MaskFraction { true -> w |> map(constantly('_')), _ -> w })) |>
map(fn(l) l |> join(' ')) |>
join('\n') |>
println()and then e.g.
pbpaste | oak masker.oak --mask-fraction 0.3Difficulty shoots through the roof for me around 30% masking. (Coincidentally, this is also where I'm training my current ContraBT5 bottleneck model.)
New latent space research updates just dropped with a Twitter thread!
DALL-E 2 prompt for physical controls appreciation.
top-down view of a high-ceilinged futuristic minimalist attic loft in SoHo, Manhattan studio, medium close-up of a hand manipulating numerous dials, buttons, and sliders on a clean white surface, digital art, color-grading, film grain, golden hour, dreamlike ethereal lighting, warm-glow basement studio vibes, still from "Interstellar"
This is the first demo I've seen that really hammered into my brain, within the next few years we're going to build software agents that are so obviously and indisputably above our human-level abilities for reasoning and problem-solving — the second-order effects are going to be inconceivable.