The Peterman Pod
Dropbox’s Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling
When you say that most people shouldn't, my immediate thought was that one of the big projects you worked on at Dropbox was migrating away from S3.
The Peterman Pod
Dropbox’s Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling
So why did Dropbox migrate away from S3?
The Peterman Pod
Dropbox’s Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling
Did S3 do any counter negotiation before you set out to leave them and say like, oh, you know, we'll cut you a deal.
The Peterman Pod
Dropbox’s Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling
Can you give an example of a tight understanding of your workload leading to like something you could do that S3 couldn't?
The Peterman Pod
Dropbox’s Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling
Yeah, when I was reading about this migration, I saw somewhere in the migration, you initially started with Go, and then the racks or something that you're running on the hardware itself was ooming too much or requesting too much memory.
The Peterman Pod
Dropbox’s Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling
And then you migrated to Rust.
The Peterman Pod
Dropbox’s Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling
The cascading failure you mentioned, was there... So that happened and then there was a postmortem.
The Peterman Pod
Dropbox’s Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling
I don't think we needed a postmortem.
The Peterman Pod
Dropbox’s Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling
When I worked on infrastructure at Instagram, we had this concept of DEFCON knobs, which are basically these configs that you could flip that would gracefully degrade your system, where you can still operate it, but maybe in the case of Dropbox, you store less replicas.
The Peterman Pod
Dropbox’s Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling
So you take on a temporary increase of risk in losing data because you need to.
The Peterman Pod
Dropbox’s Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling
Did you have something like that and did you flip it in that type of case never when it came to durability?
The Peterman Pod
Dropbox’s Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling
I see.
The Peterman Pod
Dropbox’s Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling
So S3 is kind of, um, it's like elastic storage.
The Peterman Pod
Dropbox’s Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling
When I looked at this project, it's such a massive migration.
The Peterman Pod
Dropbox’s Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling
And my first thought is how do you coordinate this whole project without breaking the system as it's running?
The Peterman Pod
Dropbox’s Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling
What are your thoughts on doing such a large migration without breaking things?
The Peterman Pod
Dropbox’s Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling
Okay, so it was double writing both systems for a while.
The Peterman Pod
Dropbox’s Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling
Okay, and then you switched over Reed's
The Peterman Pod
Dropbox’s Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling
You mentioned simplicity, and intuition-wise, it makes sense.
The Peterman Pod
Dropbox’s Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling
Do you have a concrete example, though?