Show notes
About our guest:Developer Advocate, teacher, and failed startup founder. Published author. Currently leading all things DevRel at Tracetest.io.Find our guest on:bento.me/adnanrahic (link to Adnan's socials)Adnan on MediumFind us on:All of our social channels are on bento.me/geekingoutAll of Adriana's social channels are on bento.me/adrianamvillelaShow Links:ViberNode.jsAngular JSJQueryJava SwingFreeCodeCampTracetestMalabiAspectoJest and AVA (JS testing tools)JUnitCypress (testing)Playwright (testing)K6 (load testing)OpenTelemetry DemoTracetest in the OpenTelemetry DemoCloud Native Live: The power of traces - why OpenTelemetry embraced trace-based testingGoogle Hipster ShopTrace-based Testing the OpenTelemetry Demo by Daniel DiasTracetest on Nomad (Adriana's GitHub)Tracetest Cypress IntegrationTranscript:ADRIANA: Hey, y'all. Welcome to Geeking Out. The podcast about all geeky aspects of software delivery DevOps Observability, reliability and everything in between. I'm your host Adriana Villela coming to you from Toronto, Canada. And geeking out with me today. And you know what? I did a really crappy job because I forgot to ask how to pronounce your last name.ADNAN: It's so difficult that you'll butcher it. I don't mind. Seriously, take a stab and try your best.ADRIANA: Okay. You're not even going to give me a hint?ADNAN: Go for it.ADRIANA: Oh, dude. That's mean!ADNAN: The hint is I am from the Balkans. So eastern Europe, That's the only hint?ADRIANA: Yeah, so exactly. I don't know how to pronounce the funky accents any more than most people don't know how to pronounce the weird brazilian accents. So basically geeking out with me, I have Adnan, whose last name I cannot pronounce. No, I'm going to try. Hang on. That's so mean.ADNAN: Hey, everyone. Yeah, my last name is pronounced "rah-hitch", which is the c with the thing on it (ć) is like a "ch" sound. So it's basically if you're eastern European, you'll basically know if you're anything else you would have no absolute clue. Also funky because my name is Adnan, which is Arabic and I'm white, so that's kind of strange as well. But yeah, that's just being Bosnian, you know? We have a lot of different cultures here in the Balkans and yeah...super, super proud of that and happy to be here as well. So I've known Adriana for a couple of years now.ADRIANA: We are KubeCon buddies!ADNAN: Yeah.ADRIANA: Yeah. I'm actually super excited to meet up with you at Kubecon in Paris.ADNAN: Same, same. It's going to be great. I've never been to Paris.ADRIANA: I went when I was a kid with my parents so I was 18 or 19, I think, at the time. And it's different when you travel with your parents. So I guess I wasn't like a kid kid, but I traveled with my parents so it is going to be a different experience compared to not traveling with my parents. But I mean when I went I enjoyed the touristy things. I'm hoping I can do some more kind of like off the beaten path in Paris.ADNAN: We need to figure out anybody that speaks French and just meet up with them so they can take us around.ADRIANA: Oooh, I speak French.ADNAN: Oh, right, Canada!ADRIANA: Canada. Yeah, my high school...okay... Canada is technically a bilingual country and I will put it in air quotes because, yes, we are a bilingual country, and you look at the packaging for all of our products, and it's bilingual, but realistically speaking, the further west you move from Québec, the less French you speak in school. So, like, it's mandatory in elementary school, but it peters off the further west you move. So I was lucky enough to have gone to high school in Ottawa, Canada's capital, which is a very bilingual city. And so my high school had, like, kind of a...they had an immersion program where they had different levels of immersion. So you could do, like, full immersion or half immersion. So in high school, I did half immersion for, like, I think, two years. So it meant, like, half of my classes were in French, and one of my classes that was in French was gym class.ADNAN: Wow, that was a smart move, right?ADRIANA: Yeah. But for me, picking up French wasn't so hard. I moved to Canada when I was ten, and picking up French wasn't terrible because I speak Portuguese. And so I found a lot of parallels between French and Portuguese.ADNAN: Latin languages are easier, right, if you speak one Latin language, picking up another one. Same with the Slavic languages here in eastern Europe. I have a friend who's from Czechia, and she picked up Bosnian in one year, and it's crazy. Having a Bosnian boyfriend helped her as well.ADRIANA: Yeah.ADNAN: I'm not going to lie, but still, it's doable, right? So if you speak any language that, any language from Russian to anything here in the Balkans, it's so similar that you can pick up a lot. So I imagine it's the same. I have actually two of my coworkers. One is Brazilian and one is Argentinian. But this Brazilian guy lives in Argentina and he speaks Spanish and Brazili- sorry. What the hell? Portuguese. He speaks Portuguese and he switches between them fluently and seamlessly. Right? So I think that's really cool.ADRIANA: That's very cool. I find, like, as a Portuguese speaker, understanding most Spanish words is fairly straightforward. So, like, when I went to Barcelona in 2019, I spoke "Portoñol", so I just threw in some words in Portuguese with a Spanish-ish accent and prayed that I was understood. And I went into a shoe shop in Barcelona, and I was able to actually buy a pair of shoes speaking, my second-rate Spanish.ADNAN: That happened to me in Slovakia. I was. I was in Slovakia with my wife a few months ago and went into a store, and we kind of figured it out. They knew zero English, for the record. We kind of had to do the Bosnian Slovakian and back and forth, and it went surprisingly well. So I'm very proud of that. I'm very happy to see that it works for you as well.ADRIANA: Yes. That's so super cool. Well, are you ready to get into some of our lightning round questions? All right. Okay, let's do this. Question number one. Are you left handed or right handed?ADNAN: I am right handed. So falling while skiing and hurting my left is not a problem. Eating with utensils is an issue because...that was a problem. So I kind of have to do this old switcheroo, but yeah.ADRIANA: Next question. Do you prefer iPhone or Android?ADNAN: I was an avid Android user up until, was it probably two, three years ago, and I switched everything over to Apple products. Phone, Macbooks, the headphones, earphones. Actually, I only have the Bose noise canceling headphones that aren't Apple. Everything else is basically Apple, which is...My wife hates it because she runs Android, so I can't airdrop her photos. So I need to buy the family package iCloud, so I need to send her links. And she doesn't want an iPhone. So yeah.ADRIANA: I feel ya. My sister is...she's an iPhone user, but her husband is like, an Android user, so then she's always like, I don't know how to get files over to him. I'm like, Google Drive?ADNAN: And then she goes like, just send me on Viber. It's like it destroys the photo. If I send you a photo, we go to some beautiful place in freaking Dubrovnik where they did the Game of Thrones thing, and I take a picture, she'll, "Just send me on Viber." Like, it's going to destroy the photo. And she doesn't get it. She's not an IT. Like, she doesn't get it. It's frustrating.ADRIANA: These are the issues that divide families. Just kidding.ADNAN: iCloud family solves everything, right?ADRIANA: Yeah, it's the age-old battle. My husband and I converted to Mac. Jeez...we were, I think, like four or five years married. Before that. We're like, "No! Windows! No Mac!" Like Linux, fine, but Windows or whatever, right? And we were BlackBerry users, and Mac was like...ahhhh...And then for funsies, I got...it was a Mac Mini. I think one of the first years, the Mac Mini came out and they were like, super cheap.I'm like, let's use this as a media server. Then we plug it in, and I'm like, ahhhh, I don't know how to use this. And then for funsies, I'm like, I'm going to get myself a MacBook Pro because it was like a Black Friday sale or whatever, which in Apple terms, Black Friday sales are like, it's the saddest sale ever because you're not getting a deep discount. But I'm like, oh, discount on Apple product. And then I started using my MacBook Pro, and I just bought myself like this Core i7 Intel machine super beefed up. And I'm like, but I want to use my Mac Pro, which isn't nearly as beefy. And it's so beautiful. That was kind of it for me, that converted me from then on and was like, yeah, I can't go back.ADNAN: It's too good. It just works.ADRIANA: Right? Yeah, it just works. I'm with you. Okay, next question. Similar vein. Mac, Windows or Linux?ADNAN: Oh, I was super into Linux up until I got my Mac. I was only running Linux and I just switched to Mac when the M1 chip got released. That was quite recent as well. So I was running full on Linux up until that. Absolutely hated Windows. Nothing against Microsoft. I think they're great as a company and great with what they're doing. And Windows as a system...it's perfectly fine. I just couldn't...it was just so horrible. But yeah, no, it is Mac all the way right now.ADRIANA: I feel you. Yeah. And the M chips are like..."MWAH!" When I went from my Intel Mac to my M1 Mac, I'm like, I don't hear a fan. My computer is not burning my lap. What's going on?ADNAN: And I'm still using the Air one, which is the small one without the fans, without the anything. And you charge it like a phone, you plug it in overnight.ADRIANA: Oh, yeah, the charger is like tiny.ADNAN: Plug it in overnight, you take it off the charger in the morning, you use it all day. You get back home, you plug it in when you go to sleep, and you never charge it. It's nuts.ADRIANA: The M1 Mac airs are so pretty. It's like a little piece of jewelry for your desk.ADNAN: And it goes with my very small hands. I have very ladylike hands. Right. The palm is super small, so it kind of fits nice. Right. But I need to get a bigger one. I need to get a proper one. It's getting a bit crowded with all of the Kubernetes stuff I need to run and demo.ADRIANA: Oh, yeah, I feel you. I went for the 14 inch instead of the 16 inch M1 Mac Pro...MacBook Pro. Because even though I like the screen size on the 16 inch, it's so much thicker compared to the Intel Macs of the same size that I was like. And I had a 15 inch Intel Mac. I was like, holy crap. It just feels too big. But to each their own. That's just my personal...ADNAN: I'm actually not against them bringing back all of the ports. It's fine that it's a bit thicker with all those ports. I'm right now connected with a dongle that's massive and it's taking up a ton of space on my table with all of these microphones and whatnot. And it would just make more sense to just have the jack. It's just like why not, right?ADRIANA: Yeah, I agree. It's nice having the HDMI back on the computer. And I think I have like an SD slot. Yeah, looking over, I have an SD slot on my Mac.ADNAN: You're a podcaster, you need that, right?ADRIANA: I know, right? Yeah, it's true, it's true. Okay, next question. Favorite programming language.ADNAN: Is Javascript even a language or is it all fake? What do you think?ADRIANA: Don't ask me about JavaScript. My opinions are as strong as using Windows as a primary machine. I don't want to offend anyone who likes either of those things, just a personal preference.ADNAN: I mean, I love it. I've used Javascript for ages. You remember when Angular JS 1.0 was released? Like Angular JS? That was like freaking ten years ago now. That's when I use Javascript on the front end, in Javascript terms old. I was around when Node was 0.12. That's when I started using Node.ADRIANA: Wow, that's old school. Dude, but I am older because I remember when Javascript came out before Angular and Node. Actually, and it was the OG Javascript that really turned me off from Javascript and then I ran away screaming.ADNAN: Oh, JQuery was terrible. JQuery was the bane of my existence. It was horrible.ADRIANA: I vaguely remember jquery, I just remember trying out a few things in Javascript and being like, "Next, please. I'll go to backend development, thank you." Ran to my Java. Now mind you, Java had like a front-end thing, I think it was called Swift (NOTE: it was Swing) and it was like a piece of crap. Also ran away.ADNAN: I would have thought that Java wouldn't be good in the front-end.ADRIANA: Yeah, it was bulky and horrible. Next question. Do you prefer dev or ops?ADNAN: Tough one, tough one. I do prefer dev. I do prefer dev, but I've been tightly linked to ops for five or so years, so it's a tough one, but still, I would still prefer dev. I don't mind doing the ops, but I think the automation part of ops I really like. The Kubernetes part of ops is painful. I'm not smart enough for that stuff. I swear to God. Networking and all of those infra Kubernetes things, I'm definitely not smart enough for that.ADRIANA: Kubernetes is definitely a test of patience. There's like so many aspects to it that I feel like you can be really good in one area of it and okay-ish in other areas. Enough to be dangerous kind of thing.ADNAN: Yeah, but then there are so many tools. There are so many things that you need to know. I have a buddy, actually, my best man. He's been doing ops...so automation and basically all things Kubernetes for years now. And dude, I ask him something and dude pulls out an answer out of thin air and then tells me something I've never heard about. And he goes like. Says something like, "Did you taint?" What are you talking about? It's like a tribal thing. It's tribal knowledge, that stuff. I don't really know where people learn all that. So...crazy.ADRIANA: I think it's usually like learning out of necessity. I find all my Kubernetes knowledge is learned out of necessity. It's like I need to figure out this thing. But hey, it means that you basically have a Kubernetes zombie apocalypse friend that if there's like the Kubernetes zombie apocalypse, you know who to call for help.ADNAN: Oh, yeah, for sure. He's the one. What's it called? He works in a platform team. And the platform team is all just developer experience, performance, reliability, uptime. So he's the person you call when shit hits the fan.ADRIANA: That's awesome. Okay, next question. Do you prefer JSON or YAML?ADNAN: Oooh...YAML. YAML.ADRIANA: YAML. Yeah, I'm Team YAML.ADNAN: Who doesn't love YAML?ADRIANA: The JSON people. So many people bitch about YAML. And yes, there are annoying things about YAML.ADNAN: I like to call myself a "full stack YAML developer".ADRIANA: I feel you. Lots of time buried in YAML. Okay, next question. Spaces or tabs? Which one do you like better?ADNAN: It's a tough one because I set up VSCode to do spaces, but I hit the tab and then it does the spaces.ADRIANA: Yes. Hey, that's how mine's set up too.ADNAN: Yeah, big brain move.ADRIANA: Yeah, I used to be like all tabs, but I don't know, I feel like YAML kind of made me do it. For some reason. It was an incentive to convert my tabs to spaces as you do. You hit tab and it turns it into spaces. Magic. Okay, two more questions. Do you prefer to consume content through video or text?ADNAN: For sure. I don't mind videos. It's just that it's either really long video. I mean, really long. What's really long nowadays? Ten minutes and above.ADRIANA: I know, right? TikTok nation.ADNAN: Three minutes and above, people think it's long nowadays. I'd say ten minutes. Above that is probably videos that I like watching just because of the content I can get out of it and learn from it. And then otherwise, just for day to day stuff, I would rather read a docs page, I would rather read a blog post than I would watch a video, which is, I've heard pretty strange people don't agree with me, so I might be. Yes.ADRIANA: Really? That's so funny. Because most...So then I have a very interesting, like, I have a skewed population then on this podcast, because most of the folks I've asked this question to all lean towards text.ADNAN: We're the old guard.ADRIANA: Yeah, I know, right?ADNAN: Born and raised. Before Facebook, before Instagram, TikTok.ADRIANA: I know. It's true. It's true. Yeah, I know. Whenever I watch old TV shows and there's, like, no cell phones and they're, like, calling from a pay phone, I'm like, "Just text them!" Obviously very tongue-in-cheek comment. But I'm like, damn, we are used to things that...the things that we're used to now, we take so much for granted.ADNAN: Oh, yeah, for sure.ADRIANA: Okay, final question. What is your superpower?ADNAN: Ooh, my amazing sense of humor.ADRIANA: Dun dun, dun.ADNAN: I didn't. That was terrible. That was terrible. I don't really know. I would say I am moderately funny. That would be it.ADRIANA: Which is a good superpower because you have to break the ice. Right? Especially when you're dealing with people, which you have to as part of your job. And so being moderately funny to kind of crack some exterior shells of grumpy. When you're interacting with certain people, I think that's a skill.ADNAN: I also like listening. So I like hearing something interesting, and I'm very particular with the people I have in my friends group where I generally like hearing something that I can relate to and also think about. So I like hearing that from people I don't know as well, because hearing something that you can think about and can motivate you or you can just have that thought, oh, that's really cool. Let me brainstorm on that and then figure out something that can benefit myself from that and then give feedback to the person and then have a conversation about that. I think that's where especially for me, I like the listening aspect of that. I think that's really cool.ADRIANA: Yeah, for sure. Totally agree. All right, well, you survived the lightning round.ADNAN: Yeah.ADRIANA: Give yourself pat on the back.ADNAN: Not that bad. You told me it's going to be horrible. It was not that bad.ADRIANA: I didn't say it was going to be horrible. Some people get nervous. It's supposed to be an icebreaker. Okay, well, now that we've cleared the lightning round, questions, why don't you tell folks about yourself? Like, what do you do?ADNAN: Yeah, I mean, the quick rundown would be that I do developer relations. So I've been doing developer relations since it was called technology evangelism. So I've been doing it for a while. And yeah, I think it is awesome that I am quite literally an influencer for tech people, which is very strange. I'm not very good at influencing or influencing or whatever the word is, but yeah, I like talking to people, I like listening and I like educating. And I think that was the reason why I moved into developer relations from being an engineer...so I was an engineer for almost half my career so far, and then was basically either do developer relations, which I ended up being very good at, or be a mediocre engineer. And I was like, yeah, let's kind of try doing the thing where I'm actually kind of better, instead of being kind of average as an engineer.So I said, yeah, what the hell? Why not let me try this. And that was back when the Medium blog was taking off. They had just launched their, what's it called? They had publications and whatnot, and they just taking off. Oh yeah, FreeCodeCamp was a thing back then. So I hosted a, I mean, still is now, but back then we were just starting out all over the world. They had local groups that were doing meetups, events, education, teaching people, and I was doing a lot of in-person education as well, just teaching courses. That's basically just because a startup I was doing early in my career was all focused on education as well, just like generating courses, just like Udemy, but for a smaller local market. And that kind of fell through.And then I just started doing education myself and then doing the FreeCodeCamp thing. And then the medium thing took off and my blog on Medium just got crazy. It took off so strangely that I got like 5,000 followers within less than a year just because of FreeCodeCamp's publication taking off and publishing. Yeah, I didn't really know what to do. This 24 year old kid, just like having a medium thing take off, what the hell? So that was super fun. And then I did a lot of FreeCodeCamp, was leading the local FreeCodeCamp community as well for a couple of years and that was when I figured, well this is fun. So I took a DevRel job and yeah, I've been doing it ever since and it's basically always been something in the Observability monitoring, log management, I'm going to say...space.I did some big data as well, one previous job, but it's always been something like this...that's, I'm going to say, a niche market that isn't really easy to figure out and that requires a lot of handholding and a lot of help from the, I'm going to say marketing support, sales and DevRel teams, especially for building products. And I had a nice foundation in product development because I had a startup early in my career, so I had a touch of, Okay, so the code you're writing isn't...it's not what you write and the code you write, it's the product that you generate. Nobody cares about the code, people care about the product you're putting out, right? So I think that mindset early in my career helped me a lot with DevRel as well. And yeah, I'm carrying that with me and I think that's the, if anybody asks me for advice regarding how to be good at developer relations, it's just you have to put out a good product and you have to be the connection between the end user and your team that builds the product and that's it.If you want to have product-led growth, if you want to build a community, if you want to build the influence of you as a person, you need to have a product that people want to use and you're the person that needs to tell people if it's usable or not. And by people I mean your team. I think that's the only golden rule, if there is any golden rule in developer relations is you have to be very realistic. I make jokes about us being influencers and us doing popularity contests and we do Only Fans for engineers. It's all jokes. I'm not serious. Those are humor, like comedy. But truthfully that's irrelevant.Truthfully, it's all about the product and it's all about you figuring out a strategy to position the product in a way that makes sense and for you to position yourself as somebody in the industry that is influential to talk about the industry or the thing your product is doing. So yeah, this got super serious super quickly.ADRIANA: But I think you make a really good point, because being in developer relations, you have to build trust between your audience...between you and your audience, right? They have to know that you're not like just some sleazy ass salesperson who's just trying to sell them on a product. They need to believe that you have something interesting to say and that, oh, by the way, I represent this product as well. And because they like what you have to say, I feel kind of, it naturally gravitates towards, oh, take a look at what they do kind of thing, right? So I do feel like there's a little bit of that.I think basically making a connection with your intended audience, right? Makes a huge difference. This is not the job for people who are extremely introverted or introverts who have no desire to put on an extroverted face for a limited period of time every day.ADNAN: Kind of, I mean, and also the core or the root of our job, developer relations. It started as sales engineering at Microsoft. Or where was it started as sales engineering 15 years ago where they figured out engineers don't really want to listen to salespeople. So they took engineers and made them into salespeople and then they figured that worked back then. And then how it evolved, it didn't really work. And then they figured out the evangelist role or the technology evangelist role, which was a thing ten years ago when I first heard about it. And then that kind of stopped working because evangelist is, nobody knows what the hell you're doing. One time, one lady asked me if I do churches. Like, I swear to God. She was like, what do you do? I was like, I'm a technology evangelist. Oh, so you do like church? I was like, no, I don't do the church. Like, what the hell? I was like, probably that's the problem. That's why they changed. It doesn't really make sense. Advocate. Developer relations and developer advocate kind of sounds more normal, I'm going to say.ADRIANA: Yeah, that's so funny. I would have never thought that. But I think us being in tech, it's not something that we would have necessarily associated with church. I could see how that...yeah...You were saying that you're working in an area that's fairly niche, right? Like Observability. And it's definitely one of those areas that has expanded a fair bit in the last several years, which is super exciting. Observability is near and dear to our hearts because I think it's really the part of, it's an evolutionary step in SRE, right?You can't be a good SRE these days without having a properly observable system. So it's very exciting what's happening in the space now. So many different innovations, and especially like where you work, right? With Tracetest, I think is very exciting because of the nature of what you guys do. If you can talk a little bit about that.ADNAN: Yeah, for sure. I mean, doubling down on that Observability as a space is massive. I mean, just take a look at Datadog and their IPO. That alone is insane with regards to how much it's needed in our industry. We're not really building monoliths anymore. Like 20 years ago, not having intricate monitoring and Observability tooling was okay. It was fine. Now, what are we building nowadays? Do you think Netflix, like Netflix is thousands and thousands of services that are all interconnected.They need to communicate, they need to do something, and they're all doing it together. How do you fix that if it's broken? So that's the thing. Observability now is, I'm going to say, at the birthplace of where it's going to be ten years from now. OpenTelemetry right now, it's getting to that stable state where it's available to actually use reliably in production. More and more of the libraries are stable. More and more, I'm going to say both the metrics and logs, not just the tracing libraries, are also getting to a stable state. And a lot of the tooling around the Observability space is defaulting to use OpenTelemetry libraries and OpenTelemetry as the standard for both ingesting data, collecting, generating, and all that fancy words that we all know.And I love that. And that's where I'm thinking. We are generating huge amounts of data with traces, with distributed tracing, because of our systems being distributed. So we need to use distributed tracing to actually get a context of what the hell is happening in our system. An API call isn't really an API call. With 200, it works. An API call is an API call that calls a queue, that calls a message bus, that calls a cache, that calls a database. Those are seven different steps in one API call.So it's not really just an API call that you need to have visibility into and see what's happening. That's when I'm seeing that people are only using all of that data currently for production environments, for figuring out when their users have problems, and how to circle back to their engineering teams to actually know exactly what went wrong. Which is awesome. But that's just step one in that whole cycle. We're talking about DevOps. We're talking about that new principle of DevOps where we have a cycle of, we have the developer pushing code to the ops person and then the ops person pushes it back because they have Observability to tell the developer what's going wrong. But that cycle isn't complete without the testing part. And that's where Tracetest, where I'm working right now, we're building a tool that taps into that DevOps cycle where you're using the Observability, so the distributed tracing and all of that telemetry you get from your system from OpenTelemetry and whatever tracing backends you're using, ranging from Datadog to Sumo Logic to all of the fancy big ones to ServiceNow as well. And you're tapping into that data to run your integration testing, your end-to-end testing, your UI testing with all of these. So you're basically enhancing all of the tests you already have. So that's what I think is super cool with Tracetest, is that it doesn't just give you the test tool to tap into OpenTelemetry and run test specs on the trace data itself, which means that you can basically run a test on every single part of your transaction that an API makes. You can say, say, oh, I want to make sure that this external API call returned 200 and there is no freaking way I can do that with any test tool right now. I have to mock stuff and I have to kind of fake stuff out. I need to figure out how it works. And I spend days on that instead of let me ping the API, get real data from the trace, I write my specs on that data, that's real data and I put that in my CI and you're done.You already have the data, like freaking use it, right? That's the magical part where we're already generating all of that data, you're already keeping the data, use it for testing as well. And I think that's where I'm going to say next groundbreaking step in this DevOps cycle is going to be where test tools are just lagging behind. So we need to figure something out and hopefully Tracetest is going to fill that gap, fill the shoes or whatever we want to say.ADRIANA: Yeah, I think that's what I love the most about the idea of trace-based testing is like you're already emitting traces, just take advantage of the data. You have data, as you said, use it. And the other thing that you mentioned, which I think is something that it seems so obvious when you say it, but it's not something that we're in the habit of doing yet, which is like viewing Observability as part of the SDLC, because everyone's like, oh, it's part of an SRE practice. Absolutely. But you can't have an observable system if you don't instrument your code. Where does application instrumentation come from? During development. And therefore it means that Observability starting way earlier in the SDLC than we care to admit, right? Shifting those conversations in that direction I think will be very important really for organizations to really make the most out of Observability, right?For starters, there's so many different aspects, but instrumentation, like getting into the habit of instrumenting your code and admitting that it starts earlier, I think is so important.ADNAN: I think the main thing that people can think of from a logical point of view is that we've been instrumenting with logs for decades. I don't see anybody complaining. Oh my God, I have to add, logging, it's just normal. Why is tracing and instrumenting your code? Why is that any different? The logical aspect of that is the exact same, with one addition where you have this context that propagates. And if you use OpenTelemetry...ADRIANA: That ties everything together. Heaven forbid!ADNAN: You're using OpenTelemetry libraries anyway, so it just does it automatically. You don't really have to be any... like, you can be an average engineer as I, and I can make it work, right? So it's not rocket science. The people that made OpenTelemetry, the maintainers, they're the rocket scientists. You can just drive the rocket, you don't really need to think about how it works, right?ADRIANA: Yeah, absolutely. Yeah, I'm definitely very excited. I think when Tracetest came on the market, I think one of the predecessors was Malabi, that I think was created by Aspecto and that was just for JavaScript, like code written in JavaScript. It's really cool that now we've got a tool like Tracetest that is language agnostic and it makes sense, right? Because OpenTelemetry...you can instrument your code using OpenTelemetry in so many different languages. So it makes sense that it's not restricted to one particular language because that way we take that full advantage of being able to really instrument across microservices that are written in different languages, right? Like you said, you don't have to futz around with different mocking and different libraries, like different testing libraries for different languages. We all speak the same language and it's called OpenTelemetry.ADNAN: That's the thing where a lot of the bottlenecks when testing is just, it takes too much time. We can't really see value from all of the time that we need to put into it. Then you have the blocker of oh, I need to know JavaScript for whatever...Jest, AVA, or whatever testing tool I need. Oh, I need to use Python for whatever test tool I need to use for Python and then JUnit or whatever Java. So you're zoned in and you're kind of blocked off. You're siloed into that environment versus using trace-based testing. And using testing in general with distributed tracing with trace test is that it doesn't care about the language, it only cares about the trace. So triggering the test itself, trace test handles that through a definition that you define.The definition could be YAML. You can just click it in the UI and it triggers a test for you. You get the traces back and then you create your specs based on the traces. Now these specs, they're language agnostic as well because they're generated with a, I'm going to say selector language, which is very similar to CSS. You basically select the span that you want to test. You say, I want this span to be equal to 200, I want this span to have a duration less than 200 milliseconds or whatever else test spec you want to add and that's it. And then based on that, if you want to integrate with any of your existing integration, I'm going to say integration testing tools. If you want to trigger from Cypress, you can do that as well, because we have this concept of a trigger where the trigger can be anything from an HTTP request, a gRPC request, a Cypress test, a Playwright test, a K6 load test.So basically anything externally that you're already using can initiate a trace test...trace-based test...as well. So whatever you're using, for whatever front end testing, UI testing, integration testing, load testing, or whatever type of testing you're using, you can add Tracetest to your integration testing. I'm going to say testing harness, and it's just going to work perfectly fine. And that's what I like about it, because the only integration points you need to care about is, okay, so Tracetest needs access to your traces, which are kept in Jaeger, Grafana, whatever you're using, and then it just needs to trigger your app. So that's the only integration point. If you have Cypress, you can trigger it with Cypress. If you have K6, you can do it with K6. So I like that modularity and that flexibility of, you can literally add it however you want and it only cares about the trace and it just works.ADRIANA: That's awesome. And the other thing worth mentioning is that trace test was integrated with the OpenTelemetry Demo recently, right? Like last fall, I want to say like just before KubeCon, North America.ADNAN: Yeah, it was a couple of months ago where I think it was...one or two of the maintainers, I can't remember. I think it was Pierre and Juliano. I actually met Juliano in person in Vienna at KCD Austria. I think it was before KubeCon in Chicago. Lovely guy, by the way. Shoutout: Juliano is awesome. Yeah, and we had a great talk about the addition of traces to the demo. It was basically him saying, oh, I literally broke the demo.I'm a maintainer, the tests passed, I merged the PR and I broke it, and I'm the maintainer. So I know how traces work. I'm very well versed in how it works. I know the ins and outs of the system, so I'm supposed to know how the system works as well. But I still managed to break it and I still managed to break it with passing tests. So that was a pretty worrying factor there. And that's when we started chatting through the GitHub issues and we said, yeah, well, I mean, let's try it, let's add it in, see how it works, add some integration tests. And then we did, and it's right now in the OpenTelemetry Demo.It's under the test directory, under Tracetest. You can check out every single service in the demo, has a dedicated set of tests and a test suite that runs basically on every PR, if I'm not mistaken. Right now, I think they did add that. Is it only on merges? But yeah, it's in the demo. If you want to take a look. What's also super cool is that as a spin off of that, we did the CNCF live, what's it called? I think cloud native Live team, we did a webinar, Whitney Lee and myself, we did a webinar just workshop showing how it works. So if you actually want to see me actually coding that, you can check that out on the CNCF Cloud Native Live podcast as well. Otherwise, just jump over to the demo. Yeah, I mean, easiest way for people to do it.ADRIANA: Yeah, I'll include that in the show notes. Yeah, I saw you posted that on LinkedIn. I was like, damn, I didn't know.ADNAN: They posted that on the landing page for OpenTelemetry. So that was quite humbling experience.ADRIANA: Yeah, that's very cool. Yeah. It's so nice to see, I think the integration of Tracetest in the OpenTelemetry Demo. And for those who aren't familiar, the OpenTelemetry Demo is based on the Google Hipster Shop. And it's basically the idea is to showcase what OpenTelemetry can do in kind of a complex-ish, sort of multi-microservice scenario written in different with microservices written in different languages. So it is not a simple application. There are a lot of moving parts and you can get it set up without too much effort, I would say locally, like using Docker Compose or if you're feeling adventurous, in Kubernetes. And so it's really cool to be able to integrate trace-based testing through Tracetest in the OpenTelemetry Demo because again, it's another piece of the puzzle, right?That's being put in, right? Really showcasing all the cool things that you can do with traces. It's not just for troubleshooting your production code, it is also for troubleshooting your development cycle, which is super exciting.ADNAN: It's normal. We've done test-driven development for years. I mean, come on, it's not a new thing because of the complexity of the systems we have now we're adding tracing to figure out what's happening. I mean, if we're already adding tracing to figure out what's happening in our development cycle, let's use that in the development cycle to also do the tests during development and then also integrate those tests in your CI, which is. I mean, it just makes sense for me. I'm not quite sure how is to put it.ADRIANA: Yeah, absolutely. I totally agree. It's one of those no-brainers, I think whenever someone comes up with a simple solution...and you know what's simple? When people are like, Oh yeah, that makes so much sense. Why didn't I think of that before? Well, there you go. Then that's when you know that this was the right thing to do, right?ADNAN: Yeah, definitely. Definitely. Yeah. The Demo is really awesome. I think it was Daniel from our team who wrote a super, I'm going to say, detailed blog post on how he actually added trace test to the demo, to the OpenTelemetry Demo. So that might also be just a reference point for people that want to get started because it's a nice reference point because if they want to actually contribute to the Demo and try it themselves, checking that blog post will actually make sure that they run the tests correctly and that any change they make, they actually won't break the Demo on merging.ADRIANA: Yeah, that's such a really great idea because I think I'm a strong believer that if you want to contribute to OpenTelemetry, you don't have to boil the ocean, you don't have to like, oh, I've got to be a contributor on the SDK or API or whatever. Something as simple as there's something in the OpenTelemetry Demo where maybe there's a feature request open. You can take that on in a language that tickles your fancy. Such a great way to get started with contributing to OpenTelemetry and now having something to make, I guess the testing a little less scary, or at least to help you understand if you break the application why that happened. I think having the trace-based test integrated in with it can be such a relief, if you will, because it's like, okay, I know where to look, I see what's going on. It's not like panic.ADNAN: You also get just system overview. When you run a test, you go, okay, so this API is going to touch all of these parts of my system, and then I actually know what it's touching. And if I know what it's touching, I know actually how to go in and either improve it, fix it, or run another test after I'm done with my development cycle. So I don't know if you're new to a project or if you're an open source contributor, you have no idea how that stuff works. You're just kind of trying to read the documentation, trying to look at the architecture, trying to figure out what API does what. And then you end up breaking something and then you don't have a test for it. To make sure that you know that it's broken is just a nightmare. Right? And then let's think about, yeah, nowadays, with remote companies, people working all over the globe, you have distributed teams, you have teams of, I don't know, eight to ten people working.How do I know what my colleague on the other side of the world is doing in another team? And whether that...it's just nuts, right? You need to have a way of having a reliable architecture overview when you're running your development. So your development cycles need to have a very nice systematic overview, and then your tests need to cover the happy paths of all of that, of the architecture that your system is. I mean, it's just freaking, I don't know, I think it's the future, more or less.ADRIANA: Yeah, absolutely. As a final plug for the power of trace-based testing, in theory, it makes sense, but to be able to actually see it in practice in a complex scenario, because when I first played around with Tracetest...I played around with Tracetest when it was barely out, I think it came out, I want to say May 2022. And I was like, let's get this to run in Nomad for fun. So I got a little example working on my own. I'm like, cool. But then my next thought was like, how does this work in a complex scenario, right? So being able to see it work in a complex scenario, I think is very opening and really shows the power of trace-based testing and what it can do for you, right?ADNAN: Yeah, exactly. And yeah, the roadmap right now is pretty extensive, so we have a lot of cool ideas that we want to start implementing. But yeah, I'm going to say the k six and the cypress things integrations regarding the triggering, those have been, for me at least, super exciting. I'm going to say most exciting just because we're getting true end to end testing. Finally, your UI test is generating traces from your browser that then triggers your backend. That's then generating traces across your entire backend. So basically you have an end-to-end test that covers your entire path of everything. And that's not something I've seen anywhere before.Right. So that's going to be really cool thing once people start using it. We just basically released it a couple of days ago. We did our announcement webinar yesterday, by the way, for the Cypress integration. Yeah, it is toasty fresh.ADRIANA: Very nice. Hot out of the oven.ADNAN: I'm just waiting to get people to start using it. I'm thinking it's going to be super cool.ADRIANA: Very nice. Well, we are coming up on time, but before we sign off, I was wondering if there's any parting words of wisdom you would like to share with our audience.ADNAN: Yeah, wisdom. Well, I'm not great on wisdom. Humor is probably...ADRIANA: Oh, how about a joke? Tell a bad joke.ADNAN: Way too grim. Let's do parting wisdom. Work out. Do stuff that you like when you're not working and work becomes easy.ADRIANA: Yes. I really like that. As a fellow workout fan, I fully support that, definitely.ADNAN: Because we're sitting around for most of the if. If you don't do something either in the morning or after work or even during your work break or whatever you want to call it in between, do crossfit, weightlifting, strength training, bouldering, biking, ice hockey...whatever you guys in Canada do. I don't know what you guys in Canada probably ice hockey. You probably skate to work. Ice skate to work.ADRIANA: Not with the weather like here in Toronto. It's been like 4C. Yeah. Yeah. I guess hockey is the thing in Canada. I am a terrible ice skater. I know, I know. I can get from A to B, and mostly stop.ADNAN: I'm a terrible skier, as you can see. One, one.ADRIANA: There you go. There you go. Oh, yeah. And you said you're a good skater.ADNAN: Yeah, I've been skating since I was, like three years old. So skating is good, but skiing totally different. You know, when you do the crossovers when you're ice skating. Because my muscle memory is ice skate. Muscle memory. With my. Tried doing it on skis. I face planted and just slid through the snow. It was not nice. It was not nice.ADRIANA: Yeah. Believe it or not, I'm a better skier than an ice skater. I went through a skiing phase, but I haven't skied for a very long time.ADNAN: Just waiting for the weekend because it's going to be cold again. So I'm just going to drive up to the mountains and do some skiing.ADRIANA: Oh, nice. See, you have snow. Toronto is like, I don't know...like, the rest of Canada is getting pummeled with snow, and we're kind of in this little island of, like, where everyone else gets snow, we get rain. So I'm like, it doesn't even feel like winter.ADNAN: Vancouver is super sad.ADRIANA: Yeah. Is it cold? Yeah, it's gotten cold right in Vancouver, which is also unusual for this time of year. For Vancouver, it's like rain. We're getting Vancouver weather. Yeah. Well, and on that fun note...Well, thank you so much, Adnan, for geeking out with me today, y'all. Don't forget to subscribe and be sure to check the show notes for additional resources and to connect with us and our guests on social media. Until next time...ADNAN: Peace out and geek out.ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet. Geeking Out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout.