Aman Sanger
speaker
350 appearances
2 recordings
1 series
first heard Sep 2024
last heard Oct 2024
Aman Sanger’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsNo recordings in the last 12 months.Older appearances are listed below; set an alert to hear about the next one.
Appearances
uh suspending a lot of disbelief and assuming like you had the know-how um and operate or or if you're saying like you have to operate with like the limited information you have now no no actually i would say you swoop in and you get all the information all the little heuristics all the little parameters all the all the parameters that define how the thing is trained and
Well, this gets into the question of like, are you really limited by compute and money or are you limited by these other things?
I think, yeah, because even with all this compute and like, you know, all the data you could collect in the world, I think you really are ultimately limited by not even ideas, but just like really good engineering. Like, even with all the capital in the world, would you really be able to assemble... Like, there aren't that many people in the world who really can, like, make the difference here.
And there's so much work that goes into research that is just, like, pure, really, really hard engineering work. As, like, a very...
kind of hand-wavy example, if you look at the original Transformer paper, you know, how much work was kind of joining together a lot of these really interesting concepts embedded in the literature versus then going in and writing all the codes, like maybe the CUDA kernels, maybe whatever else, I don't know if it ran on GPUs or TPUs originally, such that it actually saturated the GPU performance, right?
Getting GNOME to go in and do all this code, right? And GNOME is like probably one of the best engineers in the world. Or maybe going a step further, like the next generation of models, having these things, like getting model parallelism to work and scaling it on like, you know, thousands of or maybe tens of thousands of like V100s, which I think GBDE3 may have been.
There's just so much engineering effort that has to go into all of these things to make it work. If you really brought that cost down to... like, you know, maybe not zero, but just made it 10X easier, made it super easy for someone with really fantastic ideas to immediately get to the version of like the new architecture they dreamed up that is like getting 50, 40% utilization on the GPUs.
I think that would just speed up research by a ton.
I think all of us believe new ideas are probably needed to get all the way there to HEI. And... All of us also probably believe there exist ways of testing out those ideas at smaller scales and being fairly confident that they'll play out.
It's just quite difficult for the labs in their current position to dedicate their very limited research and engineering talent to exploring all these other ideas when there's this core thing that will probably improve performance for some decent amount of time.
I really like that point about, it feels like a lot of the time with programming, they're
two ways you can go about it one is like you think really hard carefully up front about the best possible way to do it and then you spend your limited time of engineering to actually implement it uh but i much prefer just getting in the code and like you know taking a crack at it seeing how it kind of lays out and then iterating really quickly on that that feels more fun um
I think different people do programming for different reasons. But I think the true, maybe like the best programmers are the ones that really love just like absolutely love programming.
For example, there are folks on our team who literally when they get back from work, they go and then they boot up Cursor and then they start coding on their side projects for the entire night and they stay up till 3 a.m. doing that. And when they're sad, they said, I just really need to code. And I think like,
You know, there's that level of programmer where like this obsession and love of programming, I think makes really the best programmers. And I think these types of people will really get into the details of how things work.
Yeah, so... I think a lot of us, well, all of us were originally Vim users.
Pure Vim, yeah. No NeoVim, just pure Vim in a terminal. And at least for myself, it was around the time that Copilot came out. So 2021. that I really wanted to try it. So I went into VS Code, the only platform, the only code editor in which it was available. And even though I really enjoyed using Vim, just the experience of Copilot with VS Code was more than good enough to convince me to switch.
And so that kind of was the default until we started working on Cursor.
Yeah, I distinctly remember there was this one conversation I had with Michael where before I hadn't thought super deeply and critically about scaling laws. And he kind of posed the question, why isn't scaling all you need or why isn't scaling going to result in massive gains in progress? And I think I went through like the stages of grief.
There is anger, denial, and then finally at the end, just thinking about it, acceptance. And I think I've been quite hopeful and optimistic about progress since. I think one thing I'll caveat is I think it also depends on like which domains you're going to see progress.
Showing 161–180 of 350 · page 9 of 18
← Previous
Next →