Menu
Sign In Search Podcasts Charts People & Topics Add Podcast API Pricing

Glauber Costa

๐Ÿ‘ค Speaker
358 total appearances

Appearances Over Time

Podcast Appearances

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

So debugging those problems become very easy. So in record time, Tiger Beetle managed to create a system based on deterministic simulation testing, which is their database. And the stories that Yoram would tell is like, look, we found this bug that would only happen if you would call fsync on a disk, and then fsync would return an incorrect result.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

So debugging those problems become very easy. So in record time, Tiger Beetle managed to create a system based on deterministic simulation testing, which is their database. And the stories that Yoram would tell is like, look, we found this bug that would only happen if you would call fsync on a disk, and then fsync would return an incorrect result.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

And at the same time, a packet would come from the network. I mean, he would describe the most complicated scenarios. Oh, and the end result of that, the simulator gives you a seed, and you type that seed into the simulator. Now you have every single step that happens to make that happen, right? So when we... This is one of the things that led us to believe, what if we try the rewrite?

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

And at the same time, a packet would come from the network. I mean, he would describe the most complicated scenarios. Oh, and the end result of that, the simulator gives you a seed, and you type that seed into the simulator. Now you have every single step that happens to make that happen, right? So when we... This is one of the things that led us to believe, what if we try the rewrite?

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

Because we knew that we could now try the rewrite using deterministic simulation testing. And then we also partnered with a company called Lantitis that offers like a...

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

Because we knew that we could now try the rewrite using deterministic simulation testing. And then we also partnered with a company called Lantitis that offers like a...

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

almost like the integration test analogy for deterministic simulation testing, which is a full system version of that, that will simulate things between machines, it will simulate the network, it will simulate hardware failures and stuff like that. So whatever bugs our simulator does not catch, Usually we just, you give it to antithesis the next day, antithesis catches the bug.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

almost like the integration test analogy for deterministic simulation testing, which is a full system version of that, that will simulate things between machines, it will simulate the network, it will simulate hardware failures and stuff like that. So whatever bugs our simulator does not catch, Usually we just, you give it to antithesis the next day, antithesis catches the bug.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

So we knew that with that, that was the missing part of the puzzle. And then with that, we will be able to actually create something that probably even surpasses the level of testing the SQLite has. But here's the catch. It is easier to do this on the rewrite because you have to write the system with this in mind from the get-go. It's not something that is easy to bolt upon.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

So we knew that with that, that was the missing part of the puzzle. And then with that, we will be able to actually create something that probably even surpasses the level of testing the SQLite has. But here's the catch. It is easier to do this on the rewrite because you have to write the system with this in mind from the get-go. It's not something that is easy to bolt upon.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

All of it. Yeah. So for people who are interested in DST, but don't want to go through the pains of just rewriting all sorts of things like that, I truly recommend taking a look at antithesis. Because, I mean, antithesis is amazing. It's the second... is the second next best thing. We use it in conjunction, it's not an either or.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

All of it. Yeah. So for people who are interested in DST, but don't want to go through the pains of just rewriting all sorts of things like that, I truly recommend taking a look at antithesis. Because, I mean, antithesis is amazing. It's the second... is the second next best thing. We use it in conjunction, it's not an either or.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

The analogy that I have is that our own DST is like unit tests, like we can run centuries of possibilities in two days. And it's very fast, all things considered. And Antithesis is like integration tests. So you wanna have both and we do have both. And Antithesis has been a great partner for us. But the problem with deterministic simulation testing

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

The analogy that I have is that our own DST is like unit tests, like we can run centuries of possibilities in two days. And it's very fast, all things considered. And Antithesis is like integration tests. So you wanna have both and we do have both. And Antithesis has been a great partner for us. But the problem with deterministic simulation testing

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

is that once you start importing other crates in Rust, for example, you have no idea what those crates are doing. Those crates are calling IO. So you kind of have to write everything. So it's not even that you have a crate for deterministic simulation testing. You might not even want to import something super simple 100%, but you don't want to import anything that does IO, that has a timer.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

is that once you start importing other crates in Rust, for example, you have no idea what those crates are doing. Those crates are calling IO. So you kind of have to write everything. So it's not even that you have a crate for deterministic simulation testing. You might not even want to import something super simple 100%, but you don't want to import anything that does IO, that has a timer.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

You want to write those things. So it's something very hard to do for a general purpose testing for a database like Tiger Beetle, for a database like Limbo. You know, it's worth it. And the scope is quite limited. Right. And you don't want to be importing crates all the time anyway. You know, Tiger Beetle is way more insane than we are.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

You want to write those things. So it's something very hard to do for a general purpose testing for a database like Tiger Beetle, for a database like Limbo. You know, it's worth it. And the scope is quite limited. Right. And you don't want to be importing crates all the time anyway. You know, Tiger Beetle is way more insane than we are.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

I mean, they have a policy that they just don't have dependencies. That's it. They write every single piece of code that we try to be a little bit more flexible. But we will not import a crate anymore. And again, it also depends because SQLite comes with a CLI.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

I mean, they have a policy that they just don't have dependencies. That's it. They write every single piece of code that we try to be a little bit more flexible. But we will not import a crate anymore. And again, it also depends because SQLite comes with a CLI.