Think your company’s too small to be hacked? Think again. Hackers don’t care how big you are, and our experts, Joel Vautour (PLATO) and Roberto Salgado (Websec), join Mike Hrycyk to break down how to get started with security testing. From PCI [Payment Card Industry Data Security Standard] compliance to phishing scams, they share real-world insights on spotting vulnerabilities, choosing the right testing approach, and how AI is reshaping the cybersecurity battle.
Can’t use the player?
Listen to this episode on Spotify (opens in new tab)
Episode Transcript:
Mike Hrycyk (00:00):
Hello everyone. Welcome to another episode of PLATO Panel Talks. I’m your host, Mike Hrycyk, and today we’re going to talk about security testing. And specifically, we’re going to center around how to get started in security testing and not how to get started as a security tester. That can be a different podcast. But let’s say you’re a company that’s never done security testing, and do you think maybe I need that, do I need that? If I do need that, how am I going to get it? So, we thought that would be an interesting conversation that not only our partner companies out there would be interested in, but also testers would still have an interest in that. So, to create a good conversation around that, I’ve brought in a couple of experts. So, Joel, can you introduce yourself?
Joel Vautour (00:38):
Hi everyone. My name is Joel Vautour. I’m the Chief Security Officer [CSO] for PLATO. I’ve been here for a couple years now. And prior to that, I was the security director for a private company for the Security Operations Center.
Mike Hrycyk (00:52):
Awesome. Great. Thanks. Welcome Joel. Alright, Roberto, tell us about yourself.
Roberto Salgado (00:56):
Thanks, Mike. I’m Roberto Salguda. I’m the CEO [Chief Executive Officer] of Websec Information Security Services. We’re a cybersecurity company in Victoria. I was on the board of ISACA Victoria. I currently run the OWASP [Open Worldwide Application Security Project] Victoria chapter. I like to stay pretty involved in cybersecurity. It’s something I’ve been interested in since age of 12 and I’ve been an organizer for AppSec PNW [Pacific Northwest Application Security Conference] and other events.
Mike Hrycyk (01:17):
Great. Thanks for joining us. So, security, it seems like you can’t go a day or an hour without seeing a headline about security in some other way or another. And breaches, the number of breaches has gone through the roof. So, I googled a bit before the conversation, and I came up with the following statement: In the past two years, from July 2023 to July 2025, there have been thousands of publicly disclosed data breaches. In the 12 months between September 2022 and September 2023, there were over 4,608 reported data breaches in the US alone. This is a big problem, and this problem isn’t going away. And at some point, we’ll talk about AI and this problem, but let’s sort of set our boundaries. What is a company’s responsibility towards data protection? Is there one? Joel, get us started.
Joel Vautour (02:04):
Well, that’s a great place to start, I think, because data protection is not just a check box for one. It’s a responsibility as far as I’m concerned. Any business today that’s collecting personal information from employees or has employee records, credit card information, anything like that, they have a duty to protect that information. They have to protect that from any unauthorized use. Obviously, they don’t want that information to get out. In practical terms, that means they’ve got to invest in controls and backups and put policies in place. And I think they also need to make their employees aware of it as well, like employee awareness training. They as well need to protect that data.
(02:45):
I think from a responsibility, if they fail to protect that data, it can result in legal consequences, the reputation of the company, obviously, loss of customer trust, which could also result in loss of revenues and loss of their critical business. So, to wrap it up, it’s just a fundamental responsibility that they need to do.
Mike Hrycyk (03:08):
Yeah, it’s a perfect way of putting it. There’s so many different facets of why we’re responsible for it. I’m going to toss you the same question, Roberto, and you can add to it. But also, I know that in Europe, they’re getting stronger and stronger legal restrictions around a company’s requirement to protect data. Are we going to see that happen here more extensively? It’s not just ethical anymore, right?
Roberto Salgado (03:31):
Yes, absolutely. We’re starting to see that in the US as well as in Canada, where there are certain laws in place meant to protect our data. So, if a company does get breached or some information is disclosed from the company, they have an obligation to report that. And that varies between the US, Canada, and Europe. And these laws are changing, but we’re starting to see that more in the US and Canada now, following in the steps of what Europe’s been doing with the data protection laws and the need to alert on these things.
Mike Hrycyk (04:03):
Alright, so, I mean, we’re a testing company, so often we are discussing testing, but just for the sake of our own conversation here, what’s the difference between data protection, data prevention and security testing? Are they the same thing? Are they synonymous, Roberto?
Roberto Salgado (04:19):
I mean, ultimately, you’re trying to protect the data, so there are some similarities there, but with data protection, we’re typically looking at the CIA triad, protecting the data’s Confidentiality, Integrity and Availability. This can typically mean encrypting the data, ensuring the data is encrypted at rest or during transmission. That’s an easy way of looking at data protection. Whereas data prevention might be more like blocking known attack paths. It might be more like having a firewall in place, multifactor authentication, and additional security consoles to prevent unauthorized access to this data.
Joel Vautour (04:53):
I think if I were to add anything, it would be that, obviously, data protection is just many different layers of defensive, preventative ways that no one can get that data. So, as explained, it could be the infrastructure of firewalls, encryption, as Roberto said. From a security testing perspective, I think validating those protections that have been put in place. I mean, a security tester could be testing code. You know, make sure that the doors are locked; you can’t get to the data. Or simulating real-world attacks to try to confirm that, again, those layers, those defensive layers are in place. So, that would be, as I said, there’s a bit of gray area of both, but they’re certainly separate in terms of their tasks to protect the data.
Mike Hrycyk (05:42):
Does everybody need security testing? And this question comes from the idea that, well, if I run a little mom and pop shop and I have an online informational system, do I have to be security tested? I mean, anyone who does credit card and saves that information, it seems kind of obvious, but big companies, small companies, companies without secure applications, moms and pops – what’s the need? Is it everyone? Does everyone have to be concerned, Joel?
Joel Vautour (06:07):
Yeah, cyber criminals, people often say that they don’t discriminate. If you have data that’s valuable, you have something that they can exploit and make money from; they will come after you. So yeah, I think the question is in terms of the answer, yes, everyone needs cybersecurity or security testing in their applications, either a small or mid-sized business. Because they’re a target, if they’re holding data that’s valuable. If you are a public-facing website or a store, customer-facing, you know, you’re purchasing through a payment process, you mentioned credit cards, you’re in the scope of a criminal. And they don’t care what size you are, and they really care about the vulnerabilities that they’re wanting to exploit and try to get that information.
Mike Hrycyk (06:52):
Anything to add there, Roberto?
Roberto Salgado (06:54):
So yeah, I think it’s a common myth that companies say we’re too small to become a target. The reality is that attackers are really opportunistic. So, in many cases, they’re just spraying the internet and seeing what they find, any low-hanging fruit. They’re not necessarily even looking at if the company’s small or big or the size of the company, they’re just seeing what they can get access to. So, certainly, small companies can be a target. Some flags I like to point out, just like if testing would be needed for the company, is like, well, are you handling any sensitive data? And sensitive data can typically be personal identifiable information, payment information, like credit card information, intellectual property, or health data, are kind of the main ones. If your systems or products like integrates with any off systems, APIs, this could be a good reason for security testing. If you’re about to launch a new product and you feature a mobile app, it’s another good reason to get some testing done. Or if your business is reliant on IT systems for its operations, then ransomware is one of the biggest things right now. Even if you’re a small company and you get ransomware’d and you just can’t operate, the IT systems are locked down, your business won’t operate. Those are all good flags of reasons why you would want to get security testing.
(08:06):
And just Mike, you mentioned the credit card information. So, another reason is that PCI [Payment Card Industry Data Security Standard] compliance would require you to do security testing if you handle credit card data. So, if you were to get breached, not only would you suffer the consequences of the breach, but there could be massive fines on top of that from the PCI council or other compliance institutions.
Mike Hrycyk (08:27):
And I think it’s risks like that that have shown the rise of Shopify. To have a third party that’s focused on that kind of security and to do that piece of the interaction so that they hold onto it. Something else you said, though, that I was thinking about, is that you said Joel is about the value of the data. And I think Roberto, you addressed that a little bit, but I’ve even been thinking about the social engineering component of phishing attacks. For a phishing person to be able to send you an email with just enough pertinent information to put the hook in, then have a conversation with you, then get you to commit resources or buy a thing that’s not real or hand over other information or whatever, it takes very minimal information to be able to do that. And the example that I come up with is now it’s becoming a concern to even put the dates you’re going to be out of office in an auto office reply, and who to talk to about that. Because that limited amount of information can make a clever phishing expert able to send an email that will get someone to commit resources to it. So, the level of valuable information is getting lower and lower and lower. And it’s being able to steal any information that can suddenly be a danger.
(09:40):
Okay, so we’ve touched on this idea a little bit, but who is required to have security testing, and I made required all caps in that. So, that’s in the legal. And so, we’ve talked about credit cards. If you process credit cards, there’s a requirement. So, the question that we’re going to come into now is who is required to have security testing? Who polices it? Is there a watchdog? Are there governing bodies? Is it just the government? Who is making sure that security testing is happening, Roberto?
Roberto Salgado (10:10):
Yeah, unfortunately, there is no official watchdog that is ensuring that the websites we use or the networks we access are secure. And we typically only find out about this after a breach has occurred. Typically, this should be reported to the Information and Privacy Commissioner of the place you reside or the business resides. And depending on the laws, they might require this to be made public. But aside from that, there is no one really watching to ensure that things, even with the regulations in place, we only find out after a breach has happened,
Mike Hrycyk (10:41):
Right, and then the public sort of becomes the police because then you lose in reputation, etc. And I guess there’s some fines in some places, but there’s no one making sure that it’s all okay.
Joel Vautour (10:54):
I think it also depends on the industry you might be in. If you’re in – as we said, we talked about PCI, that’s mostly credit card information consumers. If you’re in the healthcare, for example, there is HIPAA [Health Insurance Portability and Accountability Act] compliance. There are privacy things in terms of Canada where we have privacy laws that are in place that government departments, for example, might have contractual obligations. So, Roberto is correct, there’s nobody out there that’s policing you, but if you are a business and doing business with others, those that you’re doing business with probably have an expectation, or you might have a contractual obligation that might police that you have certain things in place. Are you SOC [System and Organization Controls] compliant or ISO [International Organization for Standardization] compliant? Are you following security guidelines that might be missed or something, allowing you to be more trustworthy when doing business with if you’re going to collect data and so on. So, I think other businesses will hold you accountable when you’re doing business, but there’s not necessarily – you said it, Mike, when there’s a breach, the public will hold you accountable for. They will let you know they don’t trust you or they won’t do business with you. So, you don’t want a breach to happen. You want to do security testing so that you’re not one of those headlines in the news that’s lost credibility.
Roberto Salgado (12:15):
Joel, you make a great point too. Through a lot of customers, or if a business wants to work with other businesses, typically SOC 2 [System and Organization Controls 2] will be a requirement, and this will be a security checklist of things they need to adhere to. Same with NIST [National Institute of Standards and Technology]. Unfortunately, what I’ve seen too is that the reality is some companies, when they’re preparing for an audit, they’ll set all these controls in place, and once the audit is over, they end up removing a lot of the controls they had because it gets in the way of business, of usability. So, I guess you could say the audit is the police in a way. But unfortunately, sometimes we do see for audits that all this security stuff gets put in place to pass the audit, but then no one really follows the processes, or it gets kind of removed after the audit’s been completed, cause it impacts the business in some negative aspect. So, just wanted to mention, yeah, I think audits are the unofficial police that we require, but it’s not always well executed, or they’ll just do things to pass the audit and then go back to how they were before.
Mike Hrycyk (13:13):
So, I mean, in the end, it sort of becomes risk adversity. The penalties, the downsides of having a breach, are so big that companies do what they think is necessary to make it not happen. And so, then what becomes necessary is for those companies to be the right level of afraid for the breach so that they do the spend and they put in the security features to make it less likely. So, ignorance means that it’ll be less security because you’re not as afraid of the consequences, so to speak. Would that be accurate, Joel?
Joel Vautour (13:48):
Well, I mean, I’m going to think of security products today that sell, and they’ll sell to your fear. As I just said earlier, you don’t want to be the top headlines of lost customer data or information. So, if you’re ignoring and you think that you’re secure because you have a lot of things in place and haven’t actually done testing, you’re wrong. Testing must be done to make sure that you’re up to date and have no vulnerabilities. As Roberto said, when you put security things in place, for example, two-factor authentication or multiple ways of getting access to something, and it might impede your business, so they might take them down, and that’s when people that will look at vulnerabilities and find their way in because you’ve opened a door somewhere else.
Mike Hrycyk (14:34):
Alright, so let’s shift our focus a little bit. So, we’ve now convinced everyone in our listenership that security testing is important. And so, let’s start talking about, well, what are our next steps? So, I’m a company; it doesn’t matter what size I am. I don’t have security testing, really. Security prevention hasn’t been on my radar so much. Can I just assign my IT manager to watch a couple of YouTube videos, and then he’s my security, he’s my CSO and my security tester? Is that all it takes, Roberto? Is that what you did?
Roberto Salgado (15:06):
I mean, sometimes that’s better than nothing, and it depends on how dedicated your IT person is to security. Realistically, the sooner you start security, the easier it will be and the more cost-effective it will be in the long run because technical debt is a real thing and not just in security, but in IT in general. So many companies we work with, especially the bigger they get, the more of a mess and complexity they have in their systems, where it’s so time-consuming, difficult or expensive to really test everything. Especially when they have such a high churn rate. They have people working in IT there for two years, and they moved to another company, and all the people we’re working with, it’s like they just started six months ago, and they have no idea. That information isn’t always passed on from role to role. So, the whole point is, I guess, the earlier you can start thinking about security, how to proactively address things, and there are lots of open source free options. You don’t always have to go through the commercial expensive software. There’s free open source alternatives you can look at. But just the earlier on you start, the easier it will be in the long run. Yeah.
Mike Hrycyk (16:08):
Joel?
Joel Vautour (16:09):
Yeah, I thought, I mean, I’ve been in it for 30-plus years, and one of them was I was an IT manager, but one thing for sure, like security testing, it’s not a hobby that you can just learn over YouTube. You can certainly – it’d be a risk. As I said, companies have budgets, right? Got to be honest here, and they’re tight, and there is do-it-yourself type content out there that can train you or even products that can help add a layer of security there. But the thing is, specializing in that skill and understanding how an attacker thinks and how systems can be exploited, how the business logic might expose some flaws or vulnerabilities. So yeah, an IT manager, I don’t think, as much as I have been an IT manager, and I think they have a broad knowledge. You know, a jack of all trades, so to speak, but when it comes to asking them to be a pen tester overnight, it’s like asking a contractor to be a structural engineer. It’s just pretty risky, to be honest. So, instead, I think you do need to focus on – security testing needs trained individuals and a mindset to understand how you can play that defensive and offensive part of your organization to protect the data.
Roberto Salgado (17:34):
I just wanted to add that some companies do offer different tiers of testing. So, you might start with something basic like OWASP Top 10 that might be a bit more cost-effective, but then as the business grows, the product grows. You could always expand into something like the Application Verification Security Standard. Both are OWASP methodologies for testing applications, except one, which you can do in three to five days; the other one might take two to three weeks or more. So, obviously, there’s differences in the depth of the testing that happens with these two methodologies, but if you’re a startup company, you’re on a tight budget, can always look at something like the OWASP Top 10, which is typically more cost-effective and will cover the main risks that you would face.
Mike Hrycyk (18:16):
So, you’ve both convinced me that something is better than nothing. Get started and get something towards you. A risk-based approach is the best way to go. And all of our testing listeners are very familiar with the risk-based approach. And so, that’s great. So, let’s say, though I’ve confirmed some budget, I’m going to hire an expert, and I think you’ve just touched on this a little bit, Roberto, but what kind of credentials are important for that person to have? As an example, I’ve had one of my juniors take a 16-month security course at a local college. Are they ready to be my expert? Are they ready to be a pen tester or what? As the hiring person, what should I be looking for? To you first, Roberto.
Roberto Salgado (18:54):
So, having run a security company for 15 years, I get asked this question a lot by our customers, and one thing I’d always like to refer them to is Google itself, which has released a guide on what to look for when getting a penetration test or security testing. And I always thought that the points they brought up were pretty spot on. So, I’ll just reference those points, I guess. But if you Google, Google Pen Test Guide, <laughter> you might be able to still find it. I think it’s still public. But basically, some of the things they talked about are whether you’re hiring a security testing company or if they do other things. Is security testing a secondary function of them? Maybe they do accounting or other stuff. Another thing you can look for is who’s doing the testing? So, a lot of companies, when they’re selling you on their services, they’ll talk about how their testers have the credentials, all these certificates, how they’ve spoken at these conferences, published these books. But then you get the switcheroo when you actually get the testing done, they’ll put a junior tester or someone else in their place. So, confirming who’s doing the testing and ensuring that that person is going to be the actual tester for your project is always a good thing to look at. And then of course there’s always the certificates. What certificates does the tester have? What education do they have? What projects have they posted on their GitHub repository, or conferences they spoke at, or papers they’ve published? How active are the community? So, you can really look at these things to try , and determine where the tester that you’ve been assigned is at.
Mike Hrycyk (20:23):
Alright. Good. Joel, anything to add?
Joel Vautour (20:24):
I think if you’re going to hire somebody like a security expert, I think real-world examples or experience are going to be key. Depending on the experience, again, probably that person is going to cost you more because of their experiences and what they can bring. But they will probably uncover a lot more kinds of vulnerabilities because of their experiences and what they’ve perhaps tested as a pen tester, you know, multiple web apps, APIs, mobile devices or cloud. The more experienced, obviously, the better. That’s in most expertise and trades. But I think additional things that make a good tester are not just going to find problems, but they’re also going to be able to help you fix them in their business. And so, you asked the question about credentials. I think it’s good to – I will not balk at people who go after certifications, and there’s GIAC for Pen Tester, and there’s Security Certified Professionals and stuff. A lot of pen testers will go after the certified ethical hacker because there’s a great knowledge base there to get you the information to give you that competency context. But if I were hiring a security expert, I would want to know that they’ve done a lot of real-world experience. And there’s content out there that helps you do that. There are courses that basically set up environments, and you are a hacker, and you go at it, and you test it, and you find vulnerabilities, you uncover them, and just to fine-tune your skills and stuff. So, a college course is great, but security testing is not just learned in the classroom. I think it’s a craft, and you’ve got to sharpen your skills out there in the real world.
Roberto Salgado (22:10):
And Mike, I just wanted to add that I think also if you are looking to hire a company, one of the most important things you could ask for is a sample report. Because ultimately this is going to be the deliverable, it’s what they’re going to hand over to you. So, you can really assess the type of findings that they include or look for, how they format the information, how they present that, but also the recommendations that they include for each vulnerability that they present. So, always asking for the sample report will give you a pretty good idea of what the testing will look like, what the deliverables will look like and so on.
Mike Hrycyk (22:42):
Okay, I agree. Although sometimes you have to hire a security outfit to help explain the report, but that just means they’re being thorough.
(22:48):
Okay. Roberto, we’ll start with you with this one. I want you to describe for me what a security testing engagement will look like. What will they need access to? What tools will they need? What will the results look like? What do I do with those results? Go!
Roberto Salgado (23:03):
Yeah, so it varies a lot depending on the type of assets being tested, but overall, the process is similar, where we typically would try to scope out the asset through a questionnaire, and this just allows us to understand the complexity of what we’ll be testing and the time requirements. We ultimately want to try and get a hundred percent coverage of what we’re testing, or as close to a hundred percent coverage of what we’re testing, just so we don’t leave any stone unturned. And that’s where the critical vulnerability was lying.
(23:32):
So, there’s different ways you can approach testing. You might’ve heard of black box, grey box, and white box, and this is just describing the level of access that you could provide the tester with. So, if you’re doing black box testing, this typically provides the tester with no information prior to the testing. So, this is more of a real-world scenario where someone finds your stuff on the internet and is trying to break in without having any context, documents, credentials, or anything like that.
(24:00):
On the other end is a white box where you can provide the tester with source code, documentation, credentials, and as much information as you can possibly give them. And this will allow them to kind of go through the testing quicker because they can understand the system better, have access to different components without having to try and reverse engineer this stuff or figure it out on their own.
(24:19):
And then in the middle is kind of the gray box testing, where you might provide them with a bit of information, like credentials, so they can access the authenticated portion of this system, they’re testing, so they can test that area as well. So, it really depends on what approach you take, and I think each approach has its own pros and cons to it. Ultimately, I think if you’re trying to find vulnerabilities and really lower your risk, then going more for a gray box/white box approach is typically better. Better value because the tester will be able to uncover more issues easier and in a shorter amount of time for testing.
(24:52):
In the real world, an attacker might have a year or unlimited time to really probe your system to test it and go at it. Sometimes, for a penetration test, we might have a week or two weeks, so we don’t have the same amount of time as an attacker realistically would to probe these systems. So, the more you can share with us and provide, it’ll make the testing easier and provide better value. So, it really depends on the approach of testing you take, what information you want to provide, but ultimately, being able to provide credentials, a bit of documentation, you would likely uncover more results than you would with a black box test.
(25:26):
And then in the reports, it should include the recommendations in terms that are easy to understand for anyone, layman’s terms, and it should be prioritized due to the severity. So, looking at the impact of the vulnerability and the likelihood of that vulnerability occurring, that will kind of help you determine the risk. And we’ll typically use the CVSS – Common Vulnerability Scoring System to determine the severity of each vulnerability and recommend addressing any critical high vulnerabilities within a week or two weeks. And then medium to low vulnerabilities might be a three-month timeframe, or you can sometimes just take those as an accepted risk. So, that kind of varies on the risk appetite of every business. But yeah.
Mike Hrycyk (26:07):
So, I’m going to stretch right into the next question just for time. So, I’ve done it, I’ve got a report. How often do I have to run a new test? Am I just done? I did it. I’m safe forever!
Joel Vautour (26:19):
No. <laughter>
Mike Hrycyk (26:19):
You can answer that one, Joel.
Joel Vautour (26:20):
Well, I mean, it’s not a one-and-done thing for sure. I think you’ve got to think of it as a regular health check. Testing, it’s got to be a hands-on process, and you keep going. So, every time you launch a new app or a new feature, even after a significant change in the infrastructure, even if you’ve onboarded a new client, just any changes at all, obviously, you should be doing it. And you should be doing, let’s just call it, I’m not going to say related to a dental checkup, but it’s like brushing your teeth. You don’t want to just do it once a year. You’re testing every part of your product all the time. So, I tell people, don’t just check a box that you’re compliant and you’ve done a test every time you’ve changed, every time you’ve done something, go at it, do a testing again. It might be at different levels of testing, but you can’t just leave it and forget it.
Mike Hrycyk (27:14):
What do you recommend to your clients, Roberto?
Roberto Salgado (27:17):
Same. Kind of the standard would be 6 to 12 months. Kind of similar to your dentist. But some of the PCI, for example, or some of the regulations or compliance requirements will require you to do a pen test, maybe more frequently, depending on the level you classify as for PCI, it might be every six months. Or like Joel was saying, every time you make a change to one of your systems, that would require further testing on those changes. It’s pretty standard to do it 6 to 12 months for most businesses. But you do have to keep in mind that certain compliance requirements might change the frequency of that.
Mike Hrycyk (27:50):
Yeah, I worked at a company a while ago that was PCI and PDSS [Payment Application Data Security Standard] compliant. So, there was a yearly, you had to do a yearly assessment. But then there was also, we ended up in lots of different discussions about whether you needed an ad hoc test, which was based on whether you went up a whole version number on your application, then you definitely had to do a full PCI compliance test. And then there were some other different rules. And I remember sitting in rooms saying, is that really a full version number? Do we really have to call it that? And it’s interesting that you’d have marketing in the room, and marketing would be yes, because if we do a full version number, then we get to market this as a new, excellent, amazing new version. But if you do it as a decimal or a second decimal version control number, then it’s not good for marketing.
Roberto Salgado (28:37):
It’s funny, the conflicts between the marketing team and the security team <laughter>.
Mike Hrycyk (28:42):
Alright, so we’re running up on time. So, I’m going to round us all out with one question, which I sort of alluded to at the start. So, the last question: how soon can I just assign AI to take care of my security testing? And conversely, you can add to it what AI’s new role is in the security landscape. So, we’ll start with you, Joel.
Joel Vautour (29:00):
Yeah, so I mean, just a couple of years ago, I recently left a company that was building security products, and AI was starting to be an assistant to security analysts, looking at vulnerabilities or looking at logs and events that occurred in an infrastructure and so on. And from a software testing perspective, automation was being put in place, and could AI help flag things? So, a skilled human can look at a workflow and understand it. I don’t think, at least not with the AI I’ve seen yet, can’t understand business processes. It doesn’t know what your business is. So, that is where I think humans are still there, but in terms of assisting using AI as a tool to look at the code and context. But I think critical reasoning and stuff like that, especially dealing with complex business logic, I think you’ve got to leave it to the people. But it’s getting smarter. I’m building small apps at home using AI, and it’s doing a great job. Does it have vulnerabilities? I’m sure it does, but from a security testing point, I think it’d be a great tool to help do the menial and repetitive tasks or regression testing and so on, helping them build a test plan. Those things from there. But who knows? I don’t know how – can’t set it and forget it, and AI is not going to be the one to solve it.
Mike Hrycyk (30:22):
So, you mean I can’t go to ChatGPT and say, Give me the state secrets of China yet? <laughter> Alright, Roberto, where’s AI in your universe?
Roberto Salgado (30:33):
I mean, AI is amazing. I use it for most things throughout my day these days. But can it replace a human pen tester? It’s not quite there yet. There are tons of companies and products coming out with AI pen tests, and they’re interesting. It’s a bit more like a next-generation vulnerability scanner at this point, where you can run a vulnerability scanner, it can look at the results and help you maybe weed out some false positives or try to exploit some of these vulnerabilities. But I mean, for any pen testing I do, I use AI, but I’m still running and orchestrating the AI. I’m not just letting it go off on its own because AI still has a lot of issues with hallucinations, with context limitations, or it gets confused easily. Sometimes it misses very obvious things. It’ll be in the error, like it’ll get an error message, and the answer is right there, and it just doesn’t register it somehow and tries to do some other thing that’s completely wrong. So, it can be really frustrating at times, as wonderful as it is. So, I mean it is getting better and better every day. Maybe in three years or five years, it’ll be at the point where it can mostly replace the human pen tester, but it still lacks that creative thinking in some cases or being able to chain multiple vulnerabilities together or certain things that humans really excel at where the AI fails. So, I think anyone doing a pen test should still be using AI to assist them with the penetration testing because the AI, you can think of it as a junior pen tester or an intern. Sometimes, they can get you 70-80% of the way there, but it just struggles with that last little bit. So, I think it’s fantastic and we should be using it for coding, for testing, for everything, but we shouldn’t just rely on it exclusively at this point.
(32:16):
I actually gave a presentation just a few weeks ago, and I’m giving a shorter version of it later today on pen testing with AI. For the presentation, I had AI solve the CTF [Capture the Flag] challenges from picoCTF, and it was pretty much able to solve all the easy ones on its own, most of the medium ones, sometimes with a bit of guidance. And then for the one hard challenge I gave it, it was able to get 80% of the way there, and then it just couldn’t get that last little bit. So, I looked into it manually and then I spent six hours myself trying to solve it, and I eventually did. So, it wasn’t an easy challenge, but this is where AI just got stuck and wasn’t able to move forward, and it did require a kind of human intervention. So yeah, it’s great, but we still have to give it a few years before it fully replaces us.
Mike Hrycyk (33:02):
What I took out of what you both just said is that humans are still more evil than AI. Good job, people.
(33:09):
The one thing that I’ve been seeing recently, and it’s sort of still security, but it’s a little different aspect, where AI is providing the bad guys a really good use of AI. One of them is in gathering information. So, if you’re going to use information to social engineer an attack, AI is really good at gathering and aggregating that information, and that’s a big time spend for bad guys. So, making them more efficient. So, that’s not great. And the other one is a really big way that we all, or many of us, used to look at phishing emails and say, Oh, I don’t have to worry about this. Look, there’s Phenglish in it or bad grammar, or it was just pretty easy to detect. And these days, the phishers are putting their messages through AI, so Grammarly, and it’s coming out with a well-constructed, good message that sounds intelligent, that doesn’t have grammar attacks. And so, you can’t in any way rely on your brain’s ability to find bad English to determine if something is phishing or not. You have to be smarter now because AI is making the phishers sound smarter. And so yeah, the new world is scary.
Roberto Salgado (34:25):
A hundred percent. It’s the kind of the scammers that are, or script kitties, if you call them, that are benefiting the most probably from AI. In particular, in social engineering. Like you said, they no longer have those spelling mistakes, grammar mistakes. Now they can produce their phish in a hundred different languages if they want, without any issue. And instantly. And also the emails themselves, they look much better because they can have AI generate the emails. But then the website that takes you to, the fake website, looks much better than it did before because, again, AI can do this so much easier now. And then aside from phishing, we’re seeing deep fakes, voice cloning, other technologies used for social engineering that are also using AI, leveraging AI, and actually being used in the real world already. I think there have been cases of deep fake videos that have actually worked, and they were able to steal millions of dollars from companies through that. So definitely.
Joel Vautour (35:17):
I think an AI is hiring a really, an intern, a really fast intern. It is smarter, it’s certainly faster, and it can be adaptive. AI can help us test faster, prioritize things faster or spot patterns that otherwise were missed. But it’s a tool and I that equipping our teams, our testers or network, anybody with the AI and they can understand it and use it as a tool and make it more effective, not that they’re going to beat the hackers, I guess there are more of them I think, than us defenders, but it’s certainly going to help. And I think turning security over entirely to AI is not there yet.
Roberto Salgado (36:00):
I think on the good side, too, is that we’re also using AI for detection and defence blue team stuff. In the same way attackers are using AI to generate these emails, defenders are now using AI to detect phishing and emails. So, it’s becoming an AI versus AI battle. I know I get a lot of spam messages on my cell phone, like Telegram, SMS and stuff like that. So, I actually wrote an AI bot that would respond to them pretending to be an elderly person and with the goal of just trying to take up as much of their time as possible. If they could take hours or days of their time, then they’re not scamming other people, and it’s just talking to my bots, then I’m like, that’s fantastic. But then the more I thought about it, I realized they’re probably using AI bots too. So, ultimately, it’s probably just an AI bot chatting with an AI bot. So yeah, the dead internet theories is here.
Mike Hrycyk (36:50):
But if you had a thousand or a million bots, then their power costs would go up cause they need that many bots to risk. Yeah. Okay. It’s a slippery slope.
(36:59):
Alright, well, thank you to my panel of experts for joining us in a really good conversation about security testing and how you need to engage with it. Thank you to our listeners for tuning in. I think this conversation has really illustrated that this is a deep topic, but it’s an important topic and that starting, getting started, is more important than being an expert first. But you need to grow because your bad guys are growing alongside of you.
(37:26):
So, if you had anything you’d like to add to the conversation, we’d love to hear your feedback, comments and questions. You can find us at @PLATOTesting on LinkedIn and Facebook or on our website. You can find links to all of our social media and websites in the episode description. If anyone out there wants to join in on one of our podcast chats or has a topic they’d like us to address, please reach out. If you’re enjoying listening to our technology-focused podcast, we’d love it if you could rate and review PLATO Panel Talks on whatever platform you’re listening on. Thank you again for listening, and we’ll talk to you again next time.