Menu
Sign In Search Podcasts Charts People & Topics Add Podcast API Pricing

Glauber Costa

๐Ÿ‘ค Speaker
358 total appearances

Appearances Over Time

Podcast Appearances

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

If you want to import a crate that does whatever crazy stuff to implement callers in the CLI, that's fine. But for the core of the database, we try not to import anything that could potentially do I.O. because we want to make sure that everything goes through the simulator.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

If you want to import a crate that does whatever crazy stuff to implement callers in the CLI, that's fine. But for the core of the database, we try not to import anything that could potentially do I.O. because we want to make sure that everything goes through the simulator.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

Yeah, so again, the simulator itself, you just write a couple of scenarios. And again, it's not truly magic because you do have to write the simulator, right? So the simulator just writes, imagine, for example, a workload, and the workload is like generate a couple of queries and then write, etc. But the simulator will include a fuzzing element to that.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

Yeah, so again, the simulator itself, you just write a couple of scenarios. And again, it's not truly magic because you do have to write the simulator, right? So the simulator just writes, imagine, for example, a workload, and the workload is like generate a couple of queries and then write, etc. But the simulator will include a fuzzing element to that.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

So we will, instead of like generate this query that I wrote in the unit test, you give it almost like a query generator and then it starts generating random queries. Then you start in the simulator itself. What happens, imagine this, if you want to write to a file in any software, you would call a operating system API, write.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

So we will, instead of like generate this query that I wrote in the unit test, you give it almost like a query generator and then it starts generating random queries. Then you start in the simulator itself. What happens, imagine this, if you want to write to a file in any software, you would call a operating system API, write.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

If we're talking about Rust, it's like a FS write, and then you write to the operating system. When you're writing software for DST, you don't do this. You have your own IO interface abstraction, and then all of your IO goes through this. So when you're running this in production mode, your abstraction for write just called the operating system write.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

If we're talking about Rust, it's like a FS write, and then you write to the operating system. When you're writing software for DST, you don't do this. You have your own IO interface abstraction, and then all of your IO goes through this. So when you're running this in production mode, your abstraction for write just called the operating system write.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

But when you're running in deterministic simulation testing mode, your abstraction for write runs the simulator code that will start injecting failures into this. And again, injecting failures in a deterministic way, because then if the query fails, because you injected a IO failure at that moment, you will be able to replay that session piece by piece.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

But when you're running in deterministic simulation testing mode, your abstraction for write runs the simulator code that will start injecting failures into this. And again, injecting failures in a deterministic way, because then if the query fails, because you injected a IO failure at that moment, you will be able to replay that session piece by piece.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

Is that the guy that, is that the guy that. It's barely inconvenient. Very easy, barely inconvenient. That's right. Super easy. I love that guy, man. Wow, wow, wow. Super easy, barely inconvenient. Wow, wow, wow.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

Is that the guy that, is that the guy that. It's barely inconvenient. Very easy, barely inconvenient. That's right. Super easy. I love that guy, man. Wow, wow, wow. Super easy, barely inconvenient. Wow, wow, wow.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

So how do you rewrite SQLite with the confidence that it will actually have the level of trust that SQLite acquired? It's super easy, barely an inconvenience, right? All you have to do is write the determinants.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

So how do you rewrite SQLite with the confidence that it will actually have the level of trust that SQLite acquired? It's super easy, barely an inconvenience, right? All you have to do is write the determinants.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

First of all, compatibility in a project like this, and this is the experience that we gained at Scylla when we were rewriting Apache Cassandra. Compatibility has to be like a one-way street, right? You don't want to shackle yourself and say, I will always... Because sometimes, for example, to implement a new feature, you have to do it in a different way.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

First of all, compatibility in a project like this, and this is the experience that we gained at Scylla when we were rewriting Apache Cassandra. Compatibility has to be like a one-way street, right? You don't want to shackle yourself and say, I will always... Because sometimes, for example, to implement a new feature, you have to do it in a different way.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

So it's usually something, hey, we're going to offer you the same feature set as SQLite. We're going to read your SQLite files. We're going to execute your SQLite code. And if you're not using any of the specific features that we have, we can generate SQLite files as well. But the moment you start using new features that are only present in your implementation, it's impossible to be compatible.

The Changelog: Software Development, Open Source
Turso is rewriting SQLite in Rust (Interview)

So it's usually something, hey, we're going to offer you the same feature set as SQLite. We're going to read your SQLite files. We're going to execute your SQLite code. And if you're not using any of the specific features that we have, we can generate SQLite files as well. But the moment you start using new features that are only present in your implementation, it's impossible to be compatible.