Very often, I
get the question asked to me: “What Is a
Penetration Test”? To make a long story
short, I usually tell people that it is one of the best ways to see where the
vulnerabilities of a business lie, and how to fix them up quickly”.
Of course,
for those that are in Cyber, know that there is a lot more that is involved
with that. I have been writing about
Penetration Testing for years, even published a book on it and a huge eBook that
is available to buy (I think its only $9.95 for a Kindle version of it).
I also have
very good friends and even business partners who are very good Penetration
Testers as well. I have learned a lot
from them, especially how they have crafted their art. But over the last couple of years, the
conversations have shifted to what it means
to bring in Generative AI into this realm of Cyber. Before the advent of this, Penetration
Testing has been done manually.
Meaning, one would
hire a company that specializes in doing this, and there would be actual human
beings involved, all the way from conducting the offensive exercises to writing
the final report to the client. But now,
Generative AI is taking root here, and people have started to question just how
dependable is it?
Well, this is
a difficult question to answer right off the cuff, as it will depend primarily upon
how people view Generative AI in general.
But to help you decide from a point of view, let us look at both the advantages
and disadvantages of it.
The
Advantages:
1)
Automation:
Conducting
an actual offensive exercise takes a lot of focus, attention, and brain power. Some of the more tasks that are involved here
can be quite repetitive, thus detracting away the human concentration that is
needed. But here, Generative AI can be
used to automate some of these tasks, thus leaving the Penetration Tester(s) to
focus on the big picture, which is finding the gaps and recommending to the client
the best way to fix them.
2)
Scenarios:
Before
the offensive exercises are executed, the Penetration Tester(s) must map out
the targets that they want to break down, from both an ethical and legal
perspective. Of course, nothing can be
done without the explicit permission of the client, and it must be written out
in detail in the contract. The primary objective
of the Penetration Tester(s) is to take the mindset of an actual
Cyberattacker. While the ones that I
personally know do a great job of doing this, sometimes extra help can be of
great use. In this regard, this is where
Generative AI can play a huge role. For
example, it can model other kinds of testing scenarios that the Penetration
Tester(s) may not have even thought of before.
3)
Cost:
At
one point in time, I was an actual reseller for a company that made a Penetration
Testing package that was completely automated.
When I met with the sales rep that
oversaw the Chicago market, I asked him what the price was for it. He said it was $50,000.00 to buy a license
for one year. When I heard that, my
mouth dropped, and I was thinking, WTF????
Who can afford that? But after he
explained to me in more detail that for just one flat fee, a company who buys
this license can run an unlimited amount of tests. This stands in stark contrast to the
Penetration Test that is done manually, and this can be as much as $30,000.00 -
$40,000 for just one test. Now, imagine,
you had to do this once a quarter? The
costs can really add up here. So yes,
$50K is a lot to put up front, this automated tool that is powered by
Generative AI can pay for itself in the end, depending upon how many times you
make use of it.
4)
Speed:
A
Penetration Test that is powered by Generative AI can run a comprehensive offensive
exercise in just a matter of a few hours, versus one that is done manually, which
can take weeks, or even months, depending upon the scope of the actual test. This is especially true for large scale
environments.
The Disadvantages:
1)
Mistakes:
Yes,
human Penetration Testers can make mistakes, but those tools that are powered
by Generative AI can make more of them, and for the worst of it, you may not
even know about it. For example, a fully
automated Penetration Testing tool may hit a target which has not received client
approval, and as a result, which could be prime time for a major lawsuit to
happen. Or worse yet, there could be a misconfiguration
in the tool itself, which could lead to a huge data leakage fiasco.
2)
Data:
Using
a tool that is powered by Generative AI sounds sexy and all, but there is a
fark side to it. You must train it, and
to do so, you need a large number of datasets in order to keep the models
optimized at all times. Even more
unglamorous, you must make sure that they are cleansed so that they do not give
the wrong output. For instance, suppose that
a fully automated tool hits on a target, and returns an output stating that no
vulnerabilities were found, and in fact, they really were some. This can be blamed on the lack of using
cleansed datasets, which caused the output to be skewed.
3)
Black
Box:
Generative
AI, and for that matter, all aspects of AI in general, such as Neural Networks,
Machine Learning, Computer Vision, are all deemed to be what is known as “Garbage
in And Garbage Out”. Meaning, whatever
you feed into the models will give you the output that you are seeking. In turn, this creates the phenomenon known as
the “Black Box”. Meaning, you can see what
goes in and what comes out, but you do not know what happens in between. Many of the AI vendors hold this close to
their chest, as these are primarily the algorithms that drive their
products. But, while it is great that a
client will get the outputs, they also want to know how the automated tool produced
all that. What would you tell them in that
case? If I were paying a large amount of
money for an automated Penetration Test, I would for sure want to know that.
4)
Cloud:
For
a company that migrates their entire IT and Network Infrastructure into Cloud,
it can be a nebulous process. Even after
the migration has been completed, it can still be complicated, depending upon how
much and what has been moved over. As a
result, an automated tool will not work well in this kind of environment, because
each Cloud deployment will vary quite a bit from one another. Therefore, if a client wants to take a Penetration
Test in such an environment, they are far better off hiring human Penetration
Testers. This is especially true for web-based
applications.
My
Thoughts on This:
So, the next
big question is: “Will Generative AI
replace human Penetration Testers”? My
answer to this is a blatant know. Human
intervention is still required, especially when it comes to evaluating the
results of the offensive exercises and conveying that into a written format to the
client. Heck, even the people who
started ChatGPT should always check the outputs to make sure they sound
realistic before sharing with anybody else.
If you are in
the market for having an actual Penetration Test being done at your business,
my first piece of advice is talk to an actual human being first to see what you
need to get done. Don’t simply spend the
$50K to buy an automated tool. As a
client, you also need to understand what is being done to your environment, how
the vulnerabilities will be found. These
can be best answered only by a real live human.
