George Bonaci

And you're going to learn something, but it might take you a year to say something conclusive. Or you can just run a bunch of experiments at an incredibly high velocity and it's kind of sloppy. And similarly, you might not learn anything because you made a bunch of mistakes or didn't fully think through how you'd measure something. So you kind of have to balance it.

And you're going to learn something, but it might take you a year to say something conclusive. Or you can just run a bunch of experiments at an incredibly high velocity and it's kind of sloppy. And similarly, you might not learn anything because you made a bunch of mistakes or didn't fully think through how you'd measure something. So you kind of have to balance it.

But in general, I'd probably be biased to run more things faster than run something perfectly.

But in general, I'd probably be biased to run more things faster than run something perfectly.

But in general, I'd probably be biased to run more things faster than run something perfectly.

Yeah, I think if you had to choose, velocity is more important. But there's the other side of that coin, which is if you're just doing a bunch of sloppy things, it doesn't matter how many things you run, you're not going to actually learn anything. What did you do that was sloppy that you wish wasn't sloppy? Um, it's a good question.

Yeah, I think if you had to choose, velocity is more important. But there's the other side of that coin, which is if you're just doing a bunch of sloppy things, it doesn't matter how many things you run, you're not going to actually learn anything. What did you do that was sloppy that you wish wasn't sloppy? Um, it's a good question.

Yeah, I think if you had to choose, velocity is more important. But there's the other side of that coin, which is if you're just doing a bunch of sloppy things, it doesn't matter how many things you run, you're not going to actually learn anything. What did you do that was sloppy that you wish wasn't sloppy? Um, it's a good question.

So I'm reminded of this one time, I won't say which company, but we had a webpage and the webpage is conversion. It was trending down and it was trending down for a long time. It was our number one channel. And it was like, Hey, what are we, what are we going to do to solve this? And there were kind of two paths.

So I'm reminded of this one time, I won't say which company, but we had a webpage and the webpage is conversion. It was trending down and it was trending down for a long time. It was our number one channel. And it was like, Hey, what are we, what are we going to do to solve this? And there were kind of two paths.

So I'm reminded of this one time, I won't say which company, but we had a webpage and the webpage is conversion. It was trending down and it was trending down for a long time. It was our number one channel. And it was like, Hey, what are we, what are we going to do to solve this? And there were kind of two paths.

There was one where we could run a bunch of individual, well-controlled experiments and understand, hey, changing this button or changing this H1 is like what's going to improve the page and it worked or it didn't. Or we could just run everything at once, kind of use our gut, kind of use our past experience and like hope that it worked.

There was one where we could run a bunch of individual, well-controlled experiments and understand, hey, changing this button or changing this H1 is like what's going to improve the page and it worked or it didn't. Or we could just run everything at once, kind of use our gut, kind of use our past experience and like hope that it worked.

There was one where we could run a bunch of individual, well-controlled experiments and understand, hey, changing this button or changing this H1 is like what's going to improve the page and it worked or it didn't. Or we could just run everything at once, kind of use our gut, kind of use our past experience and like hope that it worked.

And we ended up going that latter route, which was definitely the sloppier route, the less rigorous experiment. It did end up working. It did end up working. Uh, so we did end up like three X-ing the webpage conversion rate, like over the course of a couple of weeks and helped us hit our number that quarter. I say that that was like a mistake because we never actually knew what did or didn't work.

And we ended up going that latter route, which was definitely the sloppier route, the less rigorous experiment. It did end up working. It did end up working. Uh, so we did end up like three X-ing the webpage conversion rate, like over the course of a couple of weeks and helped us hit our number that quarter. I say that that was like a mistake because we never actually knew what did or didn't work.

And we ended up going that latter route, which was definitely the sloppier route, the less rigorous experiment. It did end up working. It did end up working. Uh, so we did end up like three X-ing the webpage conversion rate, like over the course of a couple of weeks and helped us hit our number that quarter. I say that that was like a mistake because we never actually knew what did or didn't work.

And we had to go back and undo a lot of the changes that we made once we did end up AB testing them. But I think that goes back to like the portfolio of bets. Like we were operating on a very short time horizon. So we had to like optimize for velocity versus like rigor.

And we had to go back and undo a lot of the changes that we made once we did end up AB testing them. But I think that goes back to like the portfolio of bets. Like we were operating on a very short time horizon. So we had to like optimize for velocity versus like rigor.

And we had to go back and undo a lot of the changes that we made once we did end up AB testing them. But I think that goes back to like the portfolio of bets. Like we were operating on a very short time horizon. So we had to like optimize for velocity versus like rigor.