Finding a bug is one thing. Convincing everyone else that it matters is another.
In this episode of PLATO Panel Talks, Mike Hrycyk sits down with Kevin Morgan (Senior Manager, PLATO) and Jason Soliven (Consultant, Zenith Point Consulting) to talk about bug advocacy—the often-overlooked skill of communicating the impact, risk, and importance of the bugs we find.
The panel digs into how to prioritize defects without treating everything as a SEV 1, how to write bugs that different stakeholders can actually understand, how to use evidence to make your case, and why knowing when to stop arguing can be just as important as knowing when to speak up.
Along the way, they share real-world examples and discuss everything from Y2K to lipstick, plus the role QA plays in helping teams make informed decisions about risk and quality.
Because being a good bug advocate isn’t about being the loudest person in the room. It’s about making sure the room understands what the bug actually means.
Can’t use the player?
Listen to this episode on Spotify (opens in a new tab)
This option provides a more reliable playback experience on mobile devices.
Episode Transcript:
Mike Hrycyk (00:00):
Hello everyone. Welcome to another episode of PLATO Panel Talks. I’m your host, Mike Hrycyk, and today we’re going to talk about bug advocacy. A little while ago, we were looking at our past episodes, and we realized we’d never done an episode on bug advocacy. Bug advocacy is a big and important thing. It’s one of the core skills that testers need to have to do their job. So, without any further ado, I would like to introduce you to our panel of experts. So Jason, tell us about yourself.
Jason Soliven (00:26):
Hi, Mike. Thanks for having me here. So, Jason Soliven, I’m a consultant. My company’s Zenith Point Consulting as well. I’ve been dev-ing since 2000 and in the QA testing game since 2006. So, I’ve been through capital enterprises, small startups, and everything in between. So yeah, I do have quite a bit of experience talking about bugs and advocating and arguing for them and things of that nature. So, glad to be here.
Mike Hrycyk (00:54):
Thanks for joining us today, Jason. Alright, Kevin, welcome back. Tell us about yourself.
Kevin Morgan (00:58):
My name’s Kevin Morgan. I’m a senior manager at PLATO. I’ve been doing quality assurance work since the early 1990s. Bug advocacy has been a big part of my career over the last 30 years, so I’m looking forward to having this discussion.
Mike Hrycyk (01:14):
Alright, so, simple question. We usually start with what is it we’re actually talking about? So, Kevin, we’ll get started with you. What’s bug advocacy?
Kevin Morgan (01:22):
So, bug advocacy is a part of the role of a quality assurance professional, where when we identify bugs, it’s not simply about identifying the bug, it’s also understanding the impact and who’s impacted, what the impact is to the end user, and making sure that bugs are defined properly and prioritized.
Mike Hrycyk (01:42):
Jason, anything to add there? Is bug advocacy just about the end user?
Jason Soliven (01:47):
Actually, bug advocacy could be a very fraught term because there’s a lot of nuance invested in that because it’s about sometimes, from the tester’s point of view, owning that bug, especially you’re the one who found the bug and you see that it’s a big impact, or you feel that it’s a big impact, into the organization or the system that you’re testing in. But there also gets to be a very political item to be the issues of discussion where some parties, depending on how big your organization that you’re testing in is, not everyone might feel the same way about the bug that you found and its impact into the system that you’re testing.
(02:21):
And then what Kevin said was true. It’s just that in every situation, a bug can be the biggest bug that’s ever been found in the organization, or it could be minuscule. It’s one of those things that mushrooms into something that no one saw two years down the road like, “oh, this bug has been there the whole time, and now it pops up to us and threatens to destroy the system? Oh!” So, yeah, there’s a lot to be discussed, which is why we have this time to dig into it.
Mike Hrycyk (02:47):
Yeah. And I think you both touched on what’s important to me is that not everyone else is necessarily from a little writeup going to understand the potential impacts and those are impacts to end users, but they’re also impacts to the system. The product owner may not be aware that if we let this happen, it’s going to corrupt our database or we’re going to have an out of error issue or out of memory issue or something like that, in the same way a dev might not understand the impact to the users and the drop-off of your customers, et cetera.
(03:15):
So, that segues nicely into the next question. With the way that we’ve trended in QA to QA not being the owner of a sign-off of a release to production and things like that, moving towards the product owner as the final word in something going and what the client and the customer can accept in terms of release, why is QA so important in bug advocacy? And we’ll go back to you first, Kevin.
Kevin Morgan (03:40):
I think when we look at it from the perspective of roles and the roles that people play in software development, the product owner is responsible for feature delivery more so than quality. So, their perspective is really that of what is the business value that’s being delivered? Whereas when you look at it from a QA perspective, we’re looking at it from not only what is the business value being delivered, but what is the quality of that business value that’s being delivered? So, does this new feature work seamlessly? Does it do what it’s expected to do? So, I think it’s still an important role for QA to play as opposed to the product owner’s role.
Mike Hrycyk (04:23):
Yeah, I agree. Jason?
Jason Soliven (04:25):
Yeah, just adding on to Kevin’s point there, especially, and I want to tie this on to the whole shift left, getting the testers and the QA folks as early as possible into the process. We could spot things as early as even in the requirements and the design phases of these implementations and these development, right? Especially bugs. The bugs that are found in requirements are worth more than the bugs that are found into the system development or even before release, because at that point that’s the cheapest, it’s time to address them. They’re the cheapest point there. A few minutes or an our hour of people’s time to really think through the requirements and say, “Hey, can this cause issues in our system down the line?”
(05:04):
And just harkening back to our last question, like think the Y2K bug. At some point in the 90s, someone during a design session said, “Well, the clock one turns 2000; this may be an issue, but maybe someone didn’t discover that and think, ‘Oh, this is not as important. We’ve got five years to deal with this.” And five years became six months, and everyone was at a panic. I think we’re all old enough to remember the news at that time because of that bug is like, oh, no one though about what happens when it goes to year 2000 and everything goes down to zero. So, that’s why if done right, we’re the first line that points this out. The challenge really for us is: how do we shine a light bright enough to make people truly understand the impact and the scale of things? And the thing is that’s a skill that we need to develop. Do we have the wherewithal and the awareness to really recognize a bug for what it is and be dispassionate about it? Take the emotions out of it, and take the emotions out of it for everyone else who would be involved in dealing with this thing.
Mike Hrycyk (05:58):
Yeah. Although, for some perspective, some of our listeners were not born at Y2K.
Jason Soliven (06:02):
Exactly.
Mike Hrycyk (06:03):
And our producer was probably still in grade school.
Jason Soliven (06:09):
I’m revealing a bit of my age there.
Mike Hrycyk (06:12):
I do think that one of the things that is sort of true about a product owner, building off of what you said, Kevin, is that their goal is the big picture. Their goal is to, I have a product, I want to release it, it needs to do the things. Some people will definitely look at all the different steps and all the different people that are involved in it, but QA spends its time, I would say, mired in the details. So, putting themselves in the shoes of particular users or looking at end-to-end of the system and understanding the impacts. So, more than anyone else, they’re going to try and see things from as many perspectives as possible, which makes us uniquely situated to be a good, strong bug advocate. And I don’t think any of us are saying no one else will advocate for bugs, especially if you’ve got a SME in the room, that SME is going to care deeply about the things that they’re interested in. But for your testers, they’re going to be interested in a whole bunch of different things.
Kevin Morgan (07:06):
I think that’s a really good point, Mike, because there are times when I actually have to work with business on the advocacy side, but also to say, “Hey, not everything’s a SEV 1.” So, you’re tamping them down. So, it’s not always QA driving the impact of everything; it’s super important. We’re also trying to look at the big picture and make sure that we get the right bugs fixed, right?
Mike Hrycyk (07:30):
Absolutely. So, I’m going to jump questions. We’ll come back to this other one in a second because you’ve segued nicely into this question. How do you advocate for a bug without becoming the person who thinks every bug is critical, which is something that every QA has wrestled with in their career? We’ll start with you this time, Jason.
Jason Soliven (07:47):
Well, in my travels, I’ve found one of the most effective tools is to really have a prioritization matrix, and it’s going to be tailored to any organization in any situation that you’re dealing with. But essentially the two things that you need to take into account for those things are the impact and have an actual definition of what a SEV 1 versus a SEV 2, 3, and 4. And this is usually outlined in the test planning. It, especially with a QA or the test lead, needs to work and agree with the clients and with the stakeholders as to what constitutes a SEV 1 versus a SEV 2. Some things are going to be crashes and taking functionalities out of play for everyone; have those clear definitions and agree to and align on them with everyone.
(08:29):
But the second thing is, and a lot of people forget about this, is how frequently, or what’s the probability, or the visibility even of a bug showing up? Because a SEV 1 bug that could crash the system that only will occur when all the dates line up to zero, that’ll only happen once, but it’ll be a big visibility to a lot of systems. But a bug that occurs, maybe a SEV 2, that occurs more frequently every time someone trips over a commonly traveled functionality can mean loss of revenue, can mean a really bad image if it’s an application or a really big headache and cost dollars for people at the end. So, you have to understand what is the definition of what a priority one bug would look like and agree with that with everyone.
(09:11):
And the second thing is be mindful of the context that you’re testing in. I’ve been in commerce applications and companies where something in the branding might be off. It may be just a cosmetic change to us, but because this is a commerce company or someone that sells a lot online, this will be a SEV 1 for them simply because anytime everyone passes through that, it’s their login page. Then it’s like, oh, wait a minute, they misspelled the brand name. That gets to be a SEV 1 if that’s going to be happening every time. But if it’s something like a regulatory firm where there is something deep in the agreements, maybe a bug in a terms and conditions, then it’s like, okay, well, for everyone else that’s very deep into the application, but that has legal implications that need to be addressed. So, being mindful of the people in the room, the context that you’re testing in, plus a really well-defined and agreed-upon prioritization matrix really will help mitigate against thinking that your bug is the biggest bug that there is and that needs to get all attention. So, that’s where I might come from.
Mike Hrycyk (10:11):
Kevin, anything to add or clarify there?
Kevin Morgan (10:13):
Yeah. So, when I work with clients, I often work with business clients as well. And one of the difficult things for them to understand is impact and how do you quantify impact of a defect? And so, quite often what I take them through is I have a workaround impact analysis piece of documentation, and it essentially says when we look at a defect, we want to know what the frequency is. We want to know who the people are that are impacted and how many people are impacted. The complexity of the workaround, if there is a workaround, if there isn’t, that’s obviously a different story. Then the time to actually implement the workaround, and again, making sure that we understand what roles are impacted. Are we talking about administrators, staffing clerks? Are we talking about secondary impacts where there’s downstream partners and stakeholders that rely on this process? And then what I teach people how to do is how to take that and turn it into a calculation of what we’re looking at as far as a defect is concerned in FTE [Full-Time Equivalent], just to manage it if it were to move into production. And I find this is a really good way also with development and product managers of saying, “Hey, this is what it’s going to cost the business if we don’t fix this defect.” Now that’s not for all defects, but it certainly is a helpful framework for people to understand what the impact of the defect is and how it’s going to impact the business if we let it go into production.
Jason Soliven (11:31):
Something that Kevin talked about rang a bell with me. Getting to understand the business impacts of bugs is a really great way to really get a fair weighting on how much attention a bug should get. But one thing I also want to stress is sometimes our role is not necessarily just to be the advocate for bugs specifically, but to sort of get a more balanced perspective on the bugs because I’ve been in rooms where product and engineering will not agree to which bugs are important to deal with, what should be dealt with first, because they’re coming in with two different lenses. And this is more like a rule of a heuristic that I work with where it’s like, don’t worry about how much work there is to resolve a bug because that kind of throws a lot of people’s lenses off. It’s like the amount of resources they throw in to fix a bug. So, it focuses on, yeah, what’s the financial impact of this bug not getting fixed? What is the technical impact of this bug getting fixed? Is there a mounting risk the longer that this bug exists? And then how do you deal with it? Sometimes they might not want to deal with it until there is a full number release, like a release one versus a release two. Or they might say, like, “Yeah, we’ll deal with it in a 1.3 because new features are being introduced that may resolve this particular issue.” So, just to the point of view, we play not necessarily sometimes as advocates, but also as an objective eye to where those issues lie and the impact to the organization, because we sometimes have to stand in between those parties who are going to be advocating for their own bugs to be fixed.
Mike Hrycyk (12:57):
I think something that you had said, well, one of you said, because I can’t remember who, was that it’s important in your test strategy and planning to have the discussions about the criteria that is important and unimportant. Another trap that people fall into, I think, is sometimes a PM or a PO will say to you, “Oh, you can’t have five criticals; make two of them severes.” And it is like, “Well, hold on, that doesn’t make any sense.” We’ve set this criteria. We understand what it is. And if you’re looking at the financial or business impact, it’s a lot easier to talk about which are important and which are unimportant. I think there is, at some point, validity in discussing how hard it will be to fix a bug. If you’re talking about I have 27 features to get out, MVP is this set. And after that we can decide, well, these are the things that we can get out and these are the things we can’t get out. And you can make decisions, but it’s still based on the needs that you have. And we should all remember that in the end, a bug advocate is not a release decider. So, we can get all the information out there, make sure that everybody understands, but then it’s not our job to argue and argue and argue. It’s to make everyone aware so that they can make the right decisions. Alright, Jason, what makes someone an effective bug advocate?
Jason Soliven (14:10):
So, my experience with the people that I’ve worked with, the best bug advocates have been the ones who understand the scale of a bug, not just to themselves, but to all of the stakeholders that’s involved in the process. And kind of ironic because being a bug advocate, it’s sort of like you, yourself, as a tester have to have a very dispassionate view on the bugs that you find versus bugs that other people find. I find that this is something that new testers and new people who are new to the QA practice fall into, where they find a bug, and they see that it’s an important bug, and they get very invested, almost at an emotional level, in making sure that this bug gets the attention because that’s how they feel that they could contribute to the project or the undertaking. So, to be an effective bug advocate, understand that you are the flashlight going into the dark places and pointing out to everyone else where the risks are, what the issues are, and being very effective at painting the accurate picture, first of all, for your team, for the engineering team, for your PM.
(15:08):
And funny that you mentioned PM earlier. If a PM is advocating for bugs based on what their priorities are and they want to make that iron triangle work – budget, schedule, scope – you have to also be an effective negotiator, but like, okay, well this is important and I know this is what’s important to you. So, I’ll work with them to then how do you present that to the higher-level stakeholders, the clients, the executive committee and things like that. So, an effective bug advocate from a QA standpoint has to be dispassionate and there should be negotiation and interpersonal skills come into play because you want to draw the right amount of light and still be aware of the nuances about how people respond to that. And as long as people are fully aware of the risks that are associated with letting a bug go or the resources involved with getting a bug fixed, then that’s basically you’ve become an invaluable part of the endeavor there.
(16:01):
I kind of like harken to an effective bug advocate would be someone like a Spock to a Kirk, to bring an Enterprise analogy. Yeah, it could get annoying, but we’re the ones who are saying that, well, if you leave this undealt with, this will be the chances, this will be the impacts and the effects of it. Now you may choose not to and take risks, but at least you’re going in there with eyes wide open.
Mike Hrycyk (16:22):
So, is Spock or Kirk the better bug advocate?
Jason Soliven (16:25):
Spock. Spock is the one who’s always on Kirk’s shoulder and Kirk is like, I want to do this thing. I want to do the thing that could be a bit risky. And Spock’s like, just so you’re aware, and very dispassionately, it’s like a 4% chance that this will go all to hell if you try to do it, but Kirk jumps and does the thing anyway. So, if we can be the Spock to the product owners or to the organization, then that’s the role that we best play.
Mike Hrycyk (16:49):
So, I’m probably going to disagree with you, but before I do that, I’m going to hand it over to Kevin to rebut if he has anything to add or…
Kevin Morgan (16:56):
Yeah, I think really the qualities that I think make a good bug advocate, one is understanding the software product that you’re dealing with, understanding not only from the perspective of what you’re testing, but the overall perspective. So, understanding different users, understanding the roles and how they interact with the system. I think that’s critically important. And the reason it’s important is because when you do the impact analysis, you have to actually think about all of the people or all of the roles that could be impacted by the particular defect. A defect may be very small to a developer, but may have an outlandish impact on users. So, I think that’s critically important. I think the other thing that’s really important is having good communication. Bug advocates have to be able to communicate what the issues are and what the impacts are effectively. And as you said, Jason, multiple times, dispassionately. And again, that’s where I think the framework helps to judge what the impact is of the defect itself and do that dispassionately.
Jason Soliven (17:53):
If I may, one thing too, I want to add to that is, and this seems to almost be cliche for anyone who’s experienced working with – you got a very passionate tester and you got a very passionate developer and the developers, it’s not uncommon for them to say, “That’s not as big of a deal. Oh, the users won’t worry about that one.” So, it’s like there’s a tendency to minimize certain items. And as a QA, we have to be not combative, but we need to be honest – we give them honest feedback and really be willing to say no and disagree with a developer or even some people on our teams. And this is one of the things when I’m especially interviewing for junior to intermediate roles, if there’s a disagreement and a dev tells you that nah, it’s not a big deal, what do you say? How do you bring evidence to bear saying, “No, but based on objective realities here, this could be a mounting risk” So, to the question of what can an effective person bring to a bug advocacy role is being able to stand up when people say, “No, that’s not as big of a deal.” And then you bring in the right amount of evidence to say, “Okay, this is why I believe this is important.” But again, if you’re talking to an executive or a PM, then it’s like, okay, well, I’ve given you everything that you need to make a decision.
Mike Hrycyk (19:04):
Excellent. What you guys said is good and I really like it, but I’m going to bring it back to the Spock and Kirk analogy there, Jason. I think that to be an effective bug advocate, you have to be able to be logical and analytical and understand the stuff so that people can see that you are thinking logically in a way that makes sense. You’re not blowing stuff up, you’re not filling it with passion and stuff so that you’re exaggerating. All that needs to be important. But if all you do is bring a list of that stuff and you talk like Spock and you’re a monotone, what you’re not doing is you’re not taking the opportunity to have people focus on the thing that they’re going to care about. Because if you’ve got three people in that room, one of them is going to maybe care about one point, the other one’s really going to care about another point. If there’s enough content there, they might not spot it. And so, as the good bug advocate, your capability is to sync well enough with an individual that you can say, this is what’s going to be important to you and to raise that importance and talk about it. But there’s also an upper limit, and you’ve both sort of touched on this, having a little bit of passion is good, having too much passion is not.
Jason Soliven (20:09):
Just a gentle pushback or just maybe a refinement because I don’t think we’re diverging on a point of view here. In my experience, what I’ve found was that this is where, I guess raising up another skill than an effective bug advocate would have, is how well did they document the bug and really dig into and if they feel that this is – again, feel with data backed up, how do they feel if this is an important bug, did they bring out all of the objective data that should ideally speak for itself enough that, okay, we’re here in this room now, these multiple parties, because this bug raised the eyebrow, at least someone here raised the bug for me as a tester. And then I guess this is where you have to understand the room that you’re in. If there is not enough attention being brought and you feel that, okay, this is important, then the passion has to be amped up to kind of bring a little bit of attention. But being mindful that if this bug affects certain aspects of the business and you’ve got a representative from different departments there, they might have different feelings and different levels of passion about fixing or not fixing a certain bug. Then being aware of what room you’re in and how this bug is going to be first perceived and how it’s going to get attention. And this is, I guess, something that would separate for a senior QA tester versus someone who’s more of an intermediate and junior is being able to understand all of those different nuances and come to a point at which, okay, I need to speak louder because no one’s hearing it versus maybe I’m just going to sit down and let the stakeholders come to an alignment on how important this bug is. And then I will try to fill in any dark spaces where there’s more data that needs to be provided for them to make a decision and so on and so forth.
Mike Hrycyk (21:49):
There’s also, and I don’t think any of us really touched on it, is a really important skill in a good bug advocate is knowing when to shut up. So, first off, you only get so much credibility when you come into a room. If you force everyone to talk for 22 minutes on a bug that’s only a level 3, you’re going to lose some of your credibility, and they’re not going to listen to you next time. If you go off the wall over-impassioned, you’re going to lose some of your credibility. So, be willing to invest more in the more important bugs and know when to just say, okay, you know what? In the end, this is a PO’s decision. So, make sure that they’re informed, that they have as good a possible way of understanding the impacts and then back off.
(22:27):
So, I think, Jason, you really just covered this question nicely. So, what I’m going to do is I’m going to throw it to Kevin to see how he wants to add to it and then you can add to it if you want. Kevin, how important is the way a bug is communicated? Can the exact same defect get a completely different response depending on how QA explains its impact? I think for this question, we’re talking about the write-up in your defect management system, not the way you stand up and talk about it. Right at the start, how do you write it up? What’s important?
Kevin Morgan (22:56):
First of all, understanding the bug. I think a lot of the time, especially junior testers, can be a little bit eager to write the bug before they completely understand the scope of it. And so, I spend a lot of time and I think it’s important that we understand the scope of the bug. So, you may be just peeling back one layer of the onion, right? With testing, it’s always good to go a little bit deeper to make sure that you have a clear understanding. And then communication, making sure that the bug is documented properly, that you have replication steps. If you can’t replicate it a bug, your developer’s going to turn around and say it’s not a bug. It can’t be replicated. I think then on top of that, you need to clearly understand what the impacts are. Again, we talked about it before, the immediate impacts, but also secondary and potentially tertiary impacts. If, say, for instance, data is to go downstream to a different partner or outside the organization, what might be those impacts? So, I think documentation and making sure that you have a clear understanding of what the problem is and making sure that that’s captured in the defect itself, in the bug, so that all parties understand. You understand what the impact is to the business value as well as what the impacts are to the different roles and to the people that are being impacted by it.
Jason Soliven (24:07):
I guess I’m landing on a certain theme here about being aware of the different people who will be affected. Kind of want to tie quickly back to sometimes if the PM has an agenda of trying to push down the number of SEV 1s so that they can get past a contract versus the actual executive who has to deal with the P&Ls [profit and loss] if a bug gets out into the open. And the theme really comes down to is knowing who the audience is, who are the different stakeholders when you’re pulling together, when you’re advocating for a bug, it’s a higher level skill to understand the impact to your immediate team, i.e. your project manager or your team lead, and then going out and saying, “Okay, what’s the impact to ultimately the end client or the customer that has to deal with it? Is this a degraded user experience? Is it a loss of functionality for them?” And then everywhere in between, so the executives, the marketing who has to deal with this, or the operations who has to put more money into fixing this bug.
(25:02):
So, I kept on thinking of bugs that I’d like, especially those intermittent ones. The ones that are so important that could crash the system, but you can’t nail down when they’re going to appear, if they’re going to appear. They may never appear in production, but at that one crucial moment where no one was thinking- at 3:00 AM- it takes the whole system down because no one thought about it, or at least everyone thought that there’s not enough resources to fix this bug and it’s like one in a million anyway, and then the million happens. That’s sort of the crucial part for someone who’s advocating for a bug to understand that if this kind of thing happens, who are the right people to pay attention to this? And ultimately, who is the decision-maker? Being aware of in the room of who actually has to release money to get this thing fixed and what are their priorities? And what are the priorities of the other people in the room? So, that’s sort of where I’m landing in terms of what’s an effective bug advocate going to look like.
Mike Hrycyk (25:55):
No, I don’t disagree with any of that. I think though, at the heart of this question is that for a bug advocate to be effective, even before the fact, it’s about the bug write-up. So, before you have someone speaking their passion in a room – in a triage room- you need to set a foundation, and that foundation is writing an effective bug, which is itself a skill that your best QAs have really honed. The first part of that is writing bugs that follow the same pattern. You’re using a defect management that you use that tool the same way so that when people look at different bugs, they form an understanding. Above and beyond that, there’s a little bit of training, making sure that the people who are going to consume that bugs understand what the different pieces of that bug mean and why they mean what they mean. And then writing a clear, well-communicated issue so people understand what you’re talking about. And for some, that is the repetition step because no matter how well you describe it, they don’t get it until they read those repetitions.
(26:52):
For complicated bugs, I’ve always liked, “Well, stop, you’ve written it, go back and try and read it again. Is it okay?” For really complicated things, get one of your colleagues to come and read it. Or these days, get AI to read it for you, saying, “Hey” – give it a persona – “Will marketing understand this bug?” AI is useful for things like that. It’s a good proofreader and a good tone checker, right? And the part where you get some variability is in the impact section of that bug. And a lot of your non-tester readers of defects like POs and marketing and business and et cetera, they might only ever read that impact section. So, make sure that impact section has the right information that will key and read for each of the different people.
Jason Soliven (27:33):
I think one of the key things too, and especially it starts out with a bug writeup, and this is how I kind of start spotting at once the really astute and really proficient testers and QOA personnel is when they see a bug, can they play out the implications of a bug if left unchecked? I don’t know if anyone here has seen the movie Margin Call, but there is a scene in the boardroom where an analyst is trying to describe some financial impacts of having extra leverage on their books for mortgage-backed securities. And the owner of the company is basically like, “Explain this to me like I was a 12-year-old boy, or a golden retriever.” And then the analyst proceeds to explain what carrying all these mortgage-backed securities on their books is going to mean for the company; essentially, it is going to bankrupt them if they don’t get them off their books. That kind of skill for a bug advocate is going to be crucial because once they see bugs and they have a lot of bugs on the plate, being able to spot the ones that are like, “You know what? This one is going to have a mounting risk of really being detrimental to the company’s operations if we don’t deal with this.” Being able to identify that in its complexity and then distill it down to something that all the other stakeholders could understand, that becomes a really crucial aspect of being an effective advocate.
Mike Hrycyk (28:48):
Awesome. Okay. We’ve used up a lot of our time. Just a couple of questions that we’re going to focus on, the hard ones. So, Kevin, tell us about a time when bug advocacy made a difference on one of your projects. Who was impacted and how?
Kevin Morgan (29:00):
One of the projects that I worked on was working for a company where we were building logistics for a new light rail that was going in. And the defects that I was seeing were technical defects. For instance, a fleet of trucks was using X amount of fuel and emitting X amount of CO2. And the young developers that I was working with didn’t really understand the idea of weighted averages and that different trucks will have different fuel consumption and they’ll have different CO2 outputs. And so, they were just tallying everything up and then dividing it by how many other trucks there were, not understanding the certain types of trucks and things like that. The other part that was difficult was one of the things we were looking at in the logistics was when a truck was stopped and for how long it was stopped and things like that.
(29:49):
And so, we were creating these dashboards, and I was going through the data and looking at dashboards, and I could see that the truck’s stoppages and sometimes the fuel usage and CO2 usage were incorrect based on the fact that they had used incorrect calculations. And so, one of the ways that I expressed it to them was when you’re doing this and when we’re doing these dashboards, there’s going to be somebody with an immense amount of power that’s looking at these things and they’re looking at these dashboards and they’re going to be looking at these trucks. And if you get the calculations incorrect and somebody looks like they’re stopped for longer than they are, that guy’s going to lose his job. And that’s important. That’s the impact that we need to talk about when we’re talking about these dashboards is that this is critical to somebody’s livelihood. And so, we need to take care that we understand what we’re actually producing here and that it’s not just a pretty graph.
(30:41):
And so, I went through and I helped them with the calculations and sort of brought them along to say, “Hey, this is the kind of thing that really matters.” And so, from an advocacy perspective, it wasn’t just a user problem at that point. It was actually a real-world situation that was going to happen in the event that we didn’t get it right. And so, I think bug advocacy can reach outside of the system as well and talk about the real-world impacts that the bugs are going to have on somebody when you get it wrong.
Mike Hrycyk (31:09):
Yeah. Great. An example from you, Jason?
Jason Soliven (31:12):
Oh man, five minutes to do this. Okay. So, not as fraught as Kevin’s example, but one of the more memorable bugs we came across was working for one of these commerce companies that sold cosmetics. I’ll decline to say any more specific than that. But one of the things they sold was lipstick. And on the website, you can buy different shades of lipstick. And one of my team members found a bug where the hex matching for the colour didn’t quite match on one of the colour options. This is kind of weird because depending on what device you’re on, you might see something different versus someone else. And which is a very important thing if we’re someone who’s buying a lipstick in a certain shade of red and you didn’t get the shade that you wanted.
(31:54):
Now it passed most, but during our user acceptance phase, one of the executives raised this, like, “This is not the right shade of red for this lipstick.” And because it was raised on the users, it was an executive, so there was a whole lot of email chain and discussion about it. And we were like, “Alright, so how did we possibly miss this?” We went through all of the devices and these things get tested, especially if they’re like commerce sites where you run through a gamut of devices. And when we talked to the executive who was testing, she was testing on an iPhone 13 mini. So, we were like, “Okay, let’s take the data that we have then on this and what proportion of their usage.” And there are metrics that measure what devices are most commonly used for these types of purchases and interactions with the website. And we saw them at that point: 0.03% of your common user base uses that particular device. So, they’ll run across this issue, with that device, on this browser and so on and so forth. And once we bought that, obviously I and my team had the sort of biases to a certain extent, but the thing is we had to step back and say, okay, let the data speak for itself and let those who are managing the project and that executive sort out what they want to do with this.
(33:07):
Obviously, cooler minds prevail, and they said, “You know what? For an iPhone 13, we’ll just have to monitor our usage.” But as long as it works for the preponderance of the devices which are going to be going into using the website, if you guys aren’t seeing that issue there, then I think we can put this bug at a P3 as opposed to a P1 like the executive was hollering at. So, that was a fun little example because it took in a lot of the things I was just talking about earlier in this call.
Mike Hrycyk (33:34):
I’ve absolutely worked on a number of companies where that would still be a P1 because it was the executive who was impacted.
Jason Soliven (33:40):
Exactly. So, committees and boards, things like that come into play. So yeah.
Mike Hrycyk (33:46):
But I mean, we all have dealt with that. But I mean, I sit here with three monitors and when I move something from one screen to the next, it changes colours and it doesn’t matter how much I fiddle with it and fix it. And I don’t care enough to get it right like a designer will.
Jason Soliven (33:59):
Exactly. A designer’s P1 bug versus an engineer’s P1 bug versus a product person’s P1 bug, not the same thing. So, we have to navigate those.
Mike Hrycyk (34:07):
Alright. To wrap up, Jason, what advice would you give someone early in their QA career who wants to become a better bug advocate?
Jason Soliven (34:16):
I guess what comes out of this call is that it’s good to be passionate about the bugs. And I understand where it comes from when you find a bug and you feel it is the most important thing, but be aware of the audience and who’s going to be affected, who the end users are. We are advocates for the users and the customers who are going to be impacted by the software systems that we produce, or we help produce. So, as long as you have that understanding and are willing to stand up to some developers who may downplay the importance of a bug, have that with you, and that’ll carry you a good chunk of the way towards progressing in your career.
Mike Hrycyk (34:50):
Alright. Kevin, same question.
Kevin Morgan (34:52):
For junior testers, understanding the risk, understanding the impact, understanding what business value is being delivered. I don’t think that being a bug advocate means you need to be the loudest person in the room. I think what you need to do is communicate the bug in a way that other people can understand why it’s important. How does it impact the customer? A lot of the time, people may not think about the secondary or tertiary impacts of a particular defect. They’ll just think, okay, no, there’s a problem with that screen. But if that screen has a trickle down effect where for instance, I think Jason mentioned it earlier, if it happens 500 times a day, that’s a different problem than something that happens once to one customer and it happens once every quarter. So, I think for young people and people that are just getting into quality assurance and understanding bug advocacy, it’s understanding really the impact, the risk, and like I said, the business value that you’re bringing.
Mike Hrycyk (35:49):
Excellent. Alright, that’s two positive answers. I’m going to give you a negative. So, one, remember that as the person who finds the bug, you are a victim of bug bias. You are going to think the things that you find are really important because you found them. You’re going to be connected to them, you’re going to be in passion. So, just be aware of that and be willing to back away from it. And then the other is know when to shut up, know when you’ve lost the battle. I used to tell my people when I was a QA manager, I would say, “Okay, here’s your job. I need you to go and you need to tell the developer about it. If they object, tell them one more time. If you’re not getting anywhere, back away because you’ll ruin that relationship. Come and tell me about it. I’m willing to stand toe to toe and yell with anyone if it’s that important to the client.” But at least half of those times, I was the person who was calming my own person down and saying, “Okay, listen, you’re not wrong, but it’s not as important because of this factor and this factor.” And so, that person with more experience helped.
Kevin Morgan (36:42):
Just wanted to add to that. I once worked with a manager that I was auditing, and it’s interesting, Jason brought up the mortgage-backed securities. I was a mortgage-backed securities officer at a financial institution, and she heard me having a conversation with one of the lenders where the lender’s mortgage could not be included as part of a mortgage-backed security because it didn’t have a good or better assessment when they came in to do an assessment on the properties. They had to have good assessments or better when they were reviewing the property. And so, I had a conversation for about 10 minutes with this lender because they really wanted it to be included in the package because they got a bonus if we could actually sell the mortgage into a pension fund.
(37:21):
And so, when the call finished, she came over to me and she said, “Hey, I heard your conversation and here’s a little piece of advice for you. When you’re going to tell somebody no, have three reasons.” And I said, “What do you mean?” And she said, “If you find one problem with the mortgage, there’s probably a couple more. And when you have that conversation with the lender, if you have three reasons, generally they’re going to capitulate.” And I think it goes the same with bugs. If you can find the reasons that something is a bug, if you can find more than one reason that it’s going to have impact, you’re more likely to get that bug through or get people to understand the severity of that bug. And this has worked really well for me over my career. And it’s not just about saying no, it’s also about saying yes. If you have the backup to say that this is a bug and here’s the three reasons that this is a bug, that’s a much more impactful statement than saying it’s a bug because the screen doesn’t work. So, I think, again, it goes back to that impact and making sure that you have your reasons lined up.
Mike Hrycyk (38:17):
Great hint. I would like to thank our panel, Kevin and Jason, for a really great discussion about bug advocacy. I think that it’s a skill that we don’t talk about enough, that we don’t train enough, that we don’t help build our people up enough, and that they just sort of grow, but it’s not a skill that we advocate for enough. So, I think it’s really good that we’ve had this conversation.
(38:36):
If you have anything you’d like to add to our conversation, we’d love to hear your feedback, comments, and questions. You can find us @PLATOTesting on LinkedIn and Facebook or on our website. You can find links to all of our social media and website in the episode description. And if anyone out there wants to join in on one of our podcast chats or has a topic that we’d like us to address, please reach out. If you are enjoying listening to our technology-focused podcast, we’d love it if you could rate and review PLATO Panel Talks on whatever platform you’re listening on. Thank you again for listening, and we’ll talk to you again next time.