Justin Johnson
speaker
345 appearances
1 recordings
1 series
first heard Nov 2025
last heard 25 Nov
Justin Johnson’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 1 in all, peaking in Nov 2025 with 1.
Appearances
but but here actually I I think there's there's a little bit of um, you know, technological confusion.
And and transformers already solved that for us, right?
Like transformers are actually not a model of sequences.
A transformer is natively a model of sets.
Um and that's very powerful.
But because um a lot of the transformers grew out of earlier architectures based around recurrent neural networks, and RNNs definitely do have like a a built-in architectural, like they do model one dimensional sequences.
Okay.
But transformers are just object uh models of sets and they can model a lot of the those sets could be, you know, one D sequences, they could be other things as well.
Yeah, yeah.
So a a transformer is actually not a model of a sequence of tokens.
A transformer is actually a model of a set of tokens.
Right.
The only thing that gives that that injects the order into it in in in the transf in the standard transformer architecture, the only thing that differentiates the order of the things is the positional embedding that you give the tokens.
But all
them all like the operators that happen inside a transformer block are either token wise, right?
So that either you have an FFN, you have QKV projections, like you have nor per token normalization.
All of those happen independently per token.
And then you have interactions between tokens through the attention mechanism.
But that's also sort of um it's it's permutation equivariant.
So if I permute my tokens, then the tension operator gets a permuted output in exactly the same way.
Showing 321–340 of 345 · page 17 of 18
← Previous
Next →