Performance Tour 2023 BBQ with Scott Moore

00:00:05] Get ready to discover the most actionable end-to-end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Colantonio, host of the Test Guild Automation Podcast, and my goal is to help you succeed with creating automation awesomeness.

[00:00:23] Joe Colantonio Hey, it's Joe, and welcome to another episode of the Test Guild performance in Site Reliability podcast. Today, for a Special Treat because we have the great Scott Moore joining us today to talk all about performance testing, his performance tour, and some secrets that he's going to reveal. So don't want to miss it. You want to stay all the way to the end. If you don't know, Scott is a Perf Evangelist, host of the performance tour at Perftour.us with over 28 years of IT experience with various platforms, and technologies, and his worked with some of the largest applications and infrastructures in the world. Scott also founded Load Tester Incorporated in 2004 and co-founded Northway Solutions Group in 2010. By in 2015, he brought his awesomeness together and created Scott Moore Consulting LLC, which is set up to provide consulting services around performance testing, performance engineering, and performance monitoring. He's an active thought leader in the performance engineering space, so he's created a lot of awesome things. And if you don't know, Scott is also the creator of such performance musical parody hits like Two 503's Ain't Bad, Technically Dumb, Black Friday My Site's Down, and much more that you need to check out his YouTube channel. There's a high-quality musical video parody geared just for performance testers and testers that you definitely have to check out. But we'll talk about that probably as well. You don't miss this episode. Check it out.

[00:01:48] This episode is brought to you by the awesome folks at SmartBear. Listen, we know load testing is tough but necessary. So investing in the right tools to automate tests, and identify bottlenecks to resolve issues fast, saves your organization both time and money. And that's why SmartBear created a LoadNinja, a sas load testing tool to help teams get full visible performance so you can release quality software faster than ever, make testing effortless and give it a shot. It's free and easy to try. Head it over to loadninja.com and learn more.

[00:02:20] Joe Colantonio Hey, Scott, welcome back to the Guild.

[00:02:25] Scott Moore Hey, Joe, it's really good to be on your show. Thank you for having me. That intro was a mouthful. I got to get a shorter bio. It's got to be a guy with loud mouth and make stupid videos here's Scott.

[00:02:36] Joe Colantonio You never give me a bio. I always use this one I've had for a while, so I even know if it's correct. So is there anything I missed in your bio that you want the Guild to know more about?

[00:02:45] Scott Moore No, no. You're pretty much right. I don't think anybody cares about that stuff anymore. They want to see the stupid videos now.

[00:02:51] Joe Colantonio So they're not stupid, they're awesome. And people need to see them. They should be going viral.

[00:02:56] Scott Moore Thank you. I've got to improve my marketing on that. No, but thanks again for having me on. I was told by Paul Grizzaffi that if I was on your show, I would be one of the cool kids again.

[00:03:08] Joe Colantonio That's good. Awesome. Always good to hear about Paul. So I guess some things you've been up to, some of these videos are part of your PERF tour, so I thought maybe we start off by, I want to touch on 2023, but before we do, maybe let's talk a little bit about what is the Perftour first for people that aren't aware of it?

[00:03:25] Scott Moore Yeah. In the late 2019, when everything was normal in our world, last I decided I wanted to go to a conference. It was going to be the Dynatrace perform conference out in Vegas and my wife said, You're going to have to figure out a way to pay for it yourself, buddy, because I'm not giving you any money. And of course, who the boss is. So I had to figure out what to do. And I thought you know what? I'll just take my phone. I'll travel from where I live in Florida to Las Vegas by car. And I know people in almost every state from Florida to Vegas. And I'll just stop and talk to people, record the conversations. Let's see if I can get a sort to pay for my trip. I sent an email out the next day and within 10 minutes I had a sponsor, which was catch point. Mehdi, the CEO of Catch Point said I'll do it. And I thought, Wow, I should have asked for way more money. That was the first thought. No, but I said, if I'm going to do it, why just stop it me holding an iPhone, talking to some friends, let's go buy some camera. So now I got to have more money because I got to buy equipment and all these things. That was the start of my content creation phase. So I drove. It took me I was on the road for about 33 days total, so I just didn't go straight to Vegas. I stopped in all these places and I went over 5500 miles in my truck and learned from vendors, from experts in the space at this conference. I mean, I talked to everybody and I was asking them the big question is where are we at with regard to performance these days and what are the problems that we need to solve? And what do I need to learn? And I learned a lot from that trip when I got back, it was right at the beginning of April 2020, which, as you know, what happened then was as soon as I got home, everything got locked down. And here I am with all this footage and I'm like, What am I going to do now? Fortunately, I had enough footage to release 17 full episodes of my journey. Part of that was with James Pulley. Part of that was on my own and with other people. And so the Performance Tour show actually just kind of came out of that. And one of the big things that I learned was we're nowhere near where I thought we were supposed to be in this day and time with regard to performance engineering, skill sets, and all that. And because of that, I'm still producing shows about it because apparently there's still a lot that needs to be done.

[00:05:45] Joe Colantonio So, you did in 2000, you did this year 2002. Were there any key takeaways like how do you know? Because you're actually speaking with some of the best of the best in performance. And usually, when you get that, you get kind of a skew result like, well, everyone's doing a performance, but what are you hearing that makes you think there's more to be done?

[00:06:02] Scott Moore Yeah, when you get out there in the field and you talk to the companies that are trying to implement some of these things that the experts and I'll point at myself have things that they're supposed to be doing. It's easier said than done. So we're just going to do the continuous performance. Oh, yeah. Just check by just use a couple of buttons and you're doing continuous performance. It's difficult and everything's becoming more complex. And when things become more complex, you don't need less testing and less stuff. You need more of it, which increases the complexity on that. And I think it's kind of unfair for someone to ever think, Oh, we're going to make our development side more complex, our systems more complex. But testing should be able to just click a couple of buttons. And so those when you get down to where people are actually doing the work, that's when you find out there's a lot of stuff that's still missing. There are still a lot of blind spots.

[00:06:51] Joe Colantonio Absolutely. It's kind of interesting this past year, I've heard more and more testers talking about performance that they need to do performance testing. So to me, it's like that's a good thing, but it almost sounds like it's almost impossible because like you said, things are getting more complicated. And maybe now they're expecting testers to have no background performance, to even know continuous performance, which is really, I mean, kind of something that's kind of a little more advanced, I would think. What are your thoughts on that?

[00:07:15] Scott Moore We're expecting a lot out of people today. I feel that there is a tendency to get away from specialization. And I think with regards to performance, that's the deal that I'm willing to die on. That's my soapbox, is to be very good at performance in all areas. You're going to need to specialize. I can't be a security penetration tester, a great functional tester, an exploratory tester, and a performance tester and be equally as good. There are things that a performance engineer needs to think about, and it's not just the testing piece, it's the engineering testing being one of those disciplines that an engineer does. A lot of things that we have to think about that aren't just about your code. They aren't just about the database, they aren't just about the infrastructure. It's everything in between, and one weak link can cause a problem. So when you're talking about performance playing a more important role and the whole shift left effort, so great to shift left and get the feedback about performance as early as possible. But we don't stop there. We still need that because there are still things that slip through the cracks and if you just have great discipline on the left side, you shift everything so far left that the whole ship capsizes to the left. You still have to have all these other disciplines in place. For example, somebody still has to pay attention to network performance. It's that might not ever be an issue for some companies, but it might be the Achilles heel for others and somebody who only thinks about their code. Well, my code is fine. Yeah, but how does your code affect the network? Is it real chatty? Are you pulling too much stuff across the network that you shouldn't be too many times? And those are things that you don't a developer isn't going to necessarily think about at the time they're writing the code. It's found out later. So that's where all these things come into play. And we're expecting a lot these days. When you hear about Site Reliability engineers, they're expected to be the find it and fix it. People who find the problem code around the solution reduce all the toil and help operations make everything a smooth-running engine. And it's like, come on. I mean, at some point you're going to start saying, well, we are saying in some of the service burnout, people are just getting overloaded and burned out and you're going to have some bad fallout from that, I think.

[00:09:30] Joe Colantonio So how should work then? I know with automation, a lot of times people have like an automation architect that sits outside sprint teams and it kind of coaches each of the sprint teams and have like an embedded tester that kind of help lead everyone into the testing effort. Do you still see performance as an activity that should be done outside of the sprint as a separate team? Or do you? I guess it depends. Or do you see it more as like a coaching feature that can then help all the different Sprint teams as they're going through their features?

[00:09:56] Scott Moore Well, I've always been a fan of teaching people how to fish. Right. But I don't expect a development team who's working on this feature, and it's a five-person team and they work in a squad, for example, say they're trying to do what Spotify did. I don't expect each individual member of that team to know as much about the performance of their code as I would. Or maybe there's one individual in that team that should be designated as you check performance, we'll write the code, and as we do code checks between us, we'll check you check for that. I would prefer to be sort of like that coach or mentor that tells them, here's your scope of performance. It's relegated to this function or feature you're working on. But I don't expect you to deal with the infrastructure issues that operations is having to do with. Focus on your area, get the feedback as early as you can to prevent it ever going downstream from you. And then downstream we have another area that will handle that. So that way, yes, it is everybody's responsibility for performance, but the scope is completely different. It's not necessarily everybody's responsibility to know everything that could go wrong performance-wise, right?

[00:11:04] Joe Colantonio Absolutely. So, Scott, we're heading into the holiday season. So this is probably released after Black Friday, but, it will be released in the holiday season, right where performance is usually all of a sudden, there's a spike in interest and what's going on, if everyone listened to the same thing they can do to prepare for holiday or unexpected spikes in their the traffic that you've seen over the years caused some of these outages?

[00:11:29] Scott Moore It's the same thing every year. I know in 2022, we are going to have news of outages and our buddy James Pulley is going to report on his show on Black Friday that he does every year. And he's going to tell everybody name and shame. I'm sure it's not even Black Friday yet. We've already seen this as we're recording this. We are fresh off of a news story where the new Taylor. Swift concert tour crashed Ticketmaster. And that's not a new thing and it's not necessarily a simple thing that has to be done. Ticketmaster's had problems for years going down because of high demand and a small period of time, and it takes somebody like Taylor Swift to just expose that problem again. So when you got these point load situations, again, they're not new. You deal with them when they come. But if that's like Ticketmaster, every time that's down, they're losing money by the minute. I mean lots of money by the minute. So I would expect their team to be more focused on that than, say, some birthday announcement website or something that's not relegated to it's all about the money that's coming in. So, yeah, they should be preparing for that. They should be preparing for scaling for that. Any retail company, if they're not concerned about compression and making sure that everything that can be as close to the user is as close as it can be with CDN and caching as much as they can cach and all the things that we know are the best practices that they're not working on that actively and haven't been for months. They're pretty much behind at this point.

[00:12:57] Joe Colantonio I definitely agree. I guess then if we're heading to 2023, people maybe don't have written in place for this year, maybe they get ready for next year, I guess, before we get into that. Are there any common themes or trends you've been seeing going into 2023 that maybe people need to know about before new technologies or techniques they can really help them with their performance efforts?

[00:13:17] Scott Moore I think you're seeing a lot more activity on the front-end performance side. There are a lot of, I would say, developer slash developer advocates that are out there. You see them a lot on Twitter talking about Google Analytics, lighthouse metrics, the page load times, they're focused on that. I think that's an area that needs to be focused on right now. So anything that you can do to make that page render faster, whether it's compression or the lowest cost you can make that front-end web page and make it as small as possible. Working within a budget or performance budget for your Web page, are the things that you should be focusing on. Those are low-hanging fruit and those are things you can see right in the browser. Anybody can open up a Chrome browser, click on developer tools and see immediately, how many things are pulling down, how large is that? How much is expensive is this web page? Whereas the back office operations they should be testing for the database, the network calls all of those typical things they would be. And I think if you focus on both of those, that's a winning combination. But the front-end piece right now, I think, has been so neglected. I'm happy to see some of these people reporting about it. And Tim Vereecke is one out of Akamai who's always talking about it. You'll see the folks from Web page test talking about it. And Catch Point owns Web page test. You'll see a lot of outages and things that happen from Catch Point. They've got a website called Websee I believe. So you can use these tools right now to find out how you're doing. And as you're going through it, gtmetrix, M-E-T-R-I-X, gtmetrix.com. Open up your front page on that site and look at it. It'll tell you everything that it sees that's wrong with it. Right. And gives you a score. Not that you have to work for 100% score, but just understand where you are in things. Are your images way too big? You're making way too many calls. These are things that are real simple best practices, but they're not necessarily talking about. When you're learning to develop the website, learning, react and JavaScript frameworks are things you're not thinking about that you're building something awesome and you want to think about that when you're coding.

[00:15:26] Joe Colantonio Absolutely. Front end. I've been seen as well. Another thing I've been seeing is more like infrastructure, like Kubernetes optimization. I think you give me the name of Stefano. Akamai?

[00:15:36] Scott Moore Akamai. Yeah, Akamai. And also Storm Forge. Both of those companies do something very similar, have a different approach to both of it. But one thing, just being back from KUBECON 2022 in Detroit, a huge theme is around efficiency. So we talk about performance efficiency. Now we're hearing about energy efficiency. Hey, your code might be fast, but if you're consuming the energy of the sun every day in your data center, that's not a good thing either. We have countries with power issues, so there's efficiency and then there's cost efficiency as well. So yeah, your stuff may be really, really fast, but it costs us three times as much as our competitor does to do the same thing. Why is that? So when you are thinking, I guess as a performance engineer, it's not just how fast something is, but how efficiently can I get that speed, and am I breaking the bank, in other words, to get good performance? That's not a good situation to be in.

[00:16:34] Joe Colantonio So a lot of these companies, when I speak to them as well, they're almost changing their language to be more Site Reliability engineering-centric. Don't know if it's my imagination. That's interesting as well. Maybe the morphing of titles or roles or responsibilities.

[00:16:47] Scott Moore Whatever Google says is the cool thing to be. Most people are we're not going to fight that and they want to be that. I don't know if that's because they think they're going to get a better salary out of it or whatever. But when you say Site Reliability engineer, some people know exactly what that means. There's a great percentage of people who have no idea what that means. What does a reliability engineer mean? We're no longer hearing about quality and performance. We're hearing about resilience and reliability. What are those terms mean to people who have been in the industry for, like you and I? 15, 20, 25, 30 years versus somebody who's in the market, you know, five years and are completely cloud native from day one, completely containerized applications that are Kubernetes first. It's the mind shift is changing. And this is something I'll mention too, also from KUBECON realizing that as a classic performance engineer, the world is changing around me. IT is changing around me. When you talk about changing terms, you hear about these tidal changes. You also hear about technology terms changing. Everything used to be APM application performance monitoring. Now it's observability. There's a reason for that, right? It's not just we're changing the name to change the name. It's we have different things to worry about now, when we were just monitoring something, it was always there. Well, now it might be their 15 minutes, and it's something else. It's morphing. So we have to make changes to what we look at and how we look at it. And so performance engineers really need to be thinking about this if they're not keeping pace with how fast this is changing. I think everybody would agree that was at KUBECON. The adoption of Kubernetes is faster than anybody ever thought it was going to be. The maturity that's bringing is faster than anybody thought. And so the vendors are having to step up to provide solutions faster than they did for previous if you want to call the movement of technologies. That's what I SAW.

[00:18:49] Joe Colantonio I must be out of the loop. This was actually the first year I heard of KUBECON, but it seems like everyone I talked to went there and they were going to and looked huge. So I guess that's one of the reasons why. Right. What's that?

[00:18:58] Scott Moore That's what all the cool kids are doing.

[00:19:02] Joe Colantonio I have it on my calendar for next year. Is that the reason why you think because a lot of people are going cloud-native and like this kind of shake-up in the industry where you need to stay ahead and that's why maybe it's something that's coming bigger?

[00:19:13] Scott Moore Well, it's the way we're building software is changing. Right. And we're going to build applications that are as flexible and hopefully as resilient as possible. So when you think about Kubernetes being able to create an environment and replicate an environment across the cloud, across pods, across clusters, being expandable at will, that there's some complexity to it, obviously, but it's the way we're building software is changing. We started out with bare metal. We went to virtual machines. The cloud vendors said, We'll take that burden away from you. Then everybody said, okay, we'll just go serverless will continue to take as much as we can away from you that you don't have to worry about managing. Just write your code and put it out here. Well, then we did that. And the next phase of that is, oh, and you know what? We have been relying on one cloud provider to provide this for us. And that's not good enough either, because every time AWS east Coast goes down, it takes 40 major companies down with it. We can't afford to have that risk. So to make it more reliable, we're going to have to deploy on all the clouds everywhere, all the time, multi-hybrid, meta, all that stuff, more complexity. So the way that we build and deploy software is continuing to change.

[00:20:24] Joe Colantonio Awesome. Alright, Scott, definitely one to touch in 2023. Before we do, let's wrap up 2022 biggest bloopers that people didn't see in 2022.

[00:20:31] Scott Moore Biggest bloopers that people didn't see. It really hasn't changed for many years before that. Still seeing the same sort of cloud dependencies. This massive shift to the cloud and the push over the last two years to the cloud has caused so many people to wake up and realize things weren't as good as they thought they were. They weren't as resilient as they thought. VPNs weren't as resilient as we thought. All these things had to be rethought of. And I think that's the big thing. And also cryptocurrency and blockchain is right now just kind of turned into a sort of a nothing burger. And there will obviously be uses for it in the future. It will come. But the big push for blockchain to be the solution for everything, I don't see it. We hear a lot about Web3, but Web3 is sort of built on top of Web2 and there's just a little bit of something that we haven't dealt with before, but a lot of it has turned into it's still kind of smoke and mirrors, so I don't see that right now as being the next great move in IT.

[00:21:33] Joe Colantonio Absolutely. And that's barbecue place you went to this year?

[00:21:36] Scott Moore Oh, by far it was in Austin, Texas, and it was called Gebbys. And it was actually a food truck. I think it was G-E-B-B-Y-S. Best brisket. I know that Texas brisket is. Legendary. But when you actually have the bomb brisket, you never forget it. So that whole different food group. Awesome.

[00:21:56] Joe Colantonio All right. So, Scott, 2023. What are your plans? are you still going to do the perf tour? Any new info? How could people toss?

[00:22:03] Scott Moore So let's talk about the performance tour. Okay. So for this year, this year of 2022 was the first year that I went from doing stuff on the side, like doing it as a fun project, creating content or blogs or whatever to being a content creator. Like I haven't done any performance testing or engineering engagements. This whole year I've been blocked off because the performance tour once a month, the podcast that I do, The SMC Journal comes out twice a week. I'm actually having to reduce that because everybody's like, Scott, you're coming out with so much stuff, we can't keep up with it. Slow down. So we're going to go to once a week next year if the sponsors renew. I'm working on that right now. But between those two shows, that keeps me completely busy. But we've raised the quality of the show as well. We're trying to make the performance tour almost like something you would see on broadcast television, major stations or cable TV. And we're getting there. The purpose of that creating all this content is to try to bring a wider audience to performance. Because let's admit it, it's kind of a dry topic unless you're geeks like us and we love this stuff. But to try to talk to a group of people who maybe are in IT or maybe they run a company, but they still don't understand the value? Performance engineers haven't done a great job of showing being able to tell the story of why they're so valuable to companies. So I'm trying to do that with yeah, we have sort of a technical interview. We have somebody on that, an expert, we talk about a topic, but then we try to provide that entertainment there. So at least they are somebody who would typically just turn the thing off, would go, Oh, hold on a second. I don't really understand everything here. I'm sort of getting it, but this guy's doing some crazy stuff. I want to see what he does next, and I think that's resonating with people. And then performance engineers that aren't the best at telling that story would say, Hey, just watch this guy what he's doing. Yeah, he's singing a song here, but look at the lyrics. This is actually true. This is my life. You need to know this. It's like, wow, I didn't realize that it was affecting us that much. So this year has been like a total transformation for me. I'm Mr. Content working with video editing software and sound and all that cameras and things, lighting. I have to worry about lighting. The good news and this is I'm going to announce it right here on your show, you'll be the first one. This is the major announcement is 2023. The performance tour show has been fully funded for filming next year, we will have two software sponsors for this and they do something very similar but different. They kind of crossover. That's as much as I can say right now. But I'll be talking about that by mid-December before the end of the year. I'll reveal who the sponsors are and it's going to be great. We're not just going to do just music videos. We're actually going to expand into some television shows that you're familiar with movie references and things still in the form of parody keeps us out of legal trouble that way. But still, we're going to be talking about really cool topics about back to the end user experience. But we might also talk about quantum computing. We might talk about the future of where we're going. So I'm excited. I hope that with the podcast if I get the renewals on the sponsors there, you'll be seeing a lot more of me talking about performance stuff for 2023 and I'll be getting out, having these meetups, hopefully hanging out with you at conferences and talking about the things I'm learning.

[00:25:27] Joe Colantonio Absolutely, Much needed and I highly recommend people do check it out. I actually caught it in Nashville with you and Scott and James Pulley. It was off the hook. So I highly recommend and we had a barbecue for free. So if you haven't been to a Perftour event, you definitely need to see Scott in 2023. So Scott, when are you going to roll out your schedule then so people can start booking their dates?

[00:25:47] Scott Moore Probably late December or 1st of January? And I'm actually formulating those plans right now. And if someone is out there watching that, is interested in hosting a meet-up somewhere right now, I'm sort of limited to the United States, but I'm not opposed to going somewhere else. I would love to take Perftour global. I'd love to go over to Europe, for example. There are things I haven't, I have never even seen, places I've never been that I want to go to, and I have friends over there as well. They love performance and they love barbecue and good food.

[00:26:19] Joe Colantonio So I love to see you in a Bollywood parody. That would be so awesome in India if you.

[00:26:24] Scott Moore Don't put it past.

[00:26:27] Joe Colantonio Okay, Scott, before we go, is there one piece of actual advice you can give to someone to help them with the performance testing efforts? And what's the best way to find the contact you or learn more about the Perftour?

[00:26:36] Joe Colantonio To find out about me, the performancetour.com, SMCJournal.com, or ScottMoore.consulting. Any of those sites will take you to me. Also very active on LinkedIn and Twitter and social media. I'm. Pretty easy to find the best content out there not to toot my own horn, but my buddies at Perfbytes, Mark Tomlinson, James Pulley, and Henrik. Those guys are great. Anything that you can go back to and look at and listen to that Perfbytes library, I actually do that myself and I learn a lot from that library. There are others obviously I'm putting together. There actually is a web page out there. I've got a spreadsheet. These are the lists of all the companies, the people that I follow, for example, on Twitter and social media, where I get my information from and I share that with others. And I'm happy to give you the link for that, Joe, so that you can share it. But those are some places that I would start. Unfortunately, I wish I could say there's a course there's a college, there are these books, there are a few books I guess I could recommend or a chapter. But this is one of those things where you got to kind of do your own research and you got to find people like me and people that are connected to me and your show, Joe, to find out this information because there's just not enough out there.

[00:27:44] Thanks again for your performance testing awesomeness. For the links of everything we value we covered in this episode. Head on over to testguildcom.kinsta.cloud/p95 and while you're there make sure to click on the try them both today link under the exclusive sponsor's section to learn all about smart bear's too awesome performance test tools solutions Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in iTunes? Reviews really do matter in the rankings of the show, and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed in creating end-to-end full-stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

[00:28:30] Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps, and automation professionals in the world, we'd love to have you join the FAM at Testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real-world value that can improve the skills or solve a problem for the Guild community. I love to hear from you head on over to testguild.info And let's make it happen.