How to Choose A Quality Pen Test

If you’ve ever asked multiple vendors for bids on a penetration test, you know side-by-side comparisons quickly break down. How can three bids for the same service span a $10,000 price range?

Clearly, qualitative differences lurk in the fine print. But how do you sort it out? Start by trusting your gut when it tells you that the low price probably has a catch. And then think about why you’re investing in a pen test in the first place. A mediocre penetration test may check a compliance box for you. But it also can leave you with a false sense of security.

“You might do phishing and a vulnerability scan and check the list for the year. But there’s so much more that goes into it,” says Senior Penetration Tester Jason Moulder. “If you’re not doing a comprehensive approach and incorporating all the elements of how an attack plays out, you don’t see the big picture.”

To make sure you’re getting a pen test that’s worth your investment, ask the following questions.

Is This Just a Glorified Vulnerability Scan?

Make sure you understand the difference between a vulnerability scan (vuln scan) and a pen test. HBS includes a vuln scan as part of its pen testing to identify misconfigurations, missing patches, etc. But a surprisingly low pen test price quote might indicate that a company plans to run only a software scan of your system rather than sending a human pen tester to test your defenses as a hacker would. Ask vendors exactly how many human hours they’re budgeting for manual testing activities such as confirming vulnerabilities, exploiting them and attempting to pivot into a breach of the larger system.

Automated scans can find only the weaknesses they’re told to look for. Plus, vuln scans can produce false positives by flagging incorrect headers or by flagging subcomponents that don’t actually compromise the overall system.

A human pen tester can vet the scan’s results for actual threats, plus explore vulnerabilities that the scan doesn’t know to look for. Real hackers use unpredictable methods that a vuln scan can’t simulate. “Just last year,” Jason says, “a few kids figured out how to bypass certain logins just by mashing keys on the keyboard. Those are the kinds of things you just won’t get outside of the human aspect.”

If a penetration testing vendor’s proposal talks mostly about the proprietary technology they use, ask for more details. That could mean that they plan to rely heavily on automated scans rather than deploying human experts to truly test your system.

Are The Pen Testers Certified AND Experienced?

Industry certifications indicate testers have a solid grounding in fundamentals such as attack life cycles. Look for titles such as Certified Ethical Hacker (CEH), Offensive Security Certified Professional (OSCP), GIAC Penetration Tester (GPEN), GIAC Certified Intrusion Analyst (GCIA) and GIAC Web Application Penetration Tester (GWAPT).

But acronyms after a person’s name don’t guarantee real-world experience. Ask for resumes of the testers who will work on your project. (Many companies keep their pen testers anonymous. But you can still ask for the resume of “Tester A.”) Some companies win your business by talking about their overall experience and then assign your project to an entry-level employee who wasn’t involved in any of the impressive projects you based your decision upon.

An experienced pen tester approaches every engagement with a high level of curiosity and creativity because that’s what a hacker will do. For example, a high-level pen tester working on a university job might research which of the school’s departments have recently received grants and target those departments for attacks. If they’re getting grants, they probably have valuable intellectual property to steal.

Make every decision with the idea that you’re building a system to stop hackers who do whatever it takes to break in. You need a savvy pen tester with the same mindset.

Will You Get a Team Approach to Your Test?

Remember how heist movies always feature teams of specialists who each step in to disable the security system, crack the safe, drive the escape car, etc.? In the same way, the best pen testing vendors assign multiple experts to your job. For a comprehensive test, you want a team that takes its best shot at your system with pros versed in software development, Internet of Things (IoT) devices, hardware and more.

What Methodologies do the Pen Testers Use?

Pen testing is both art and science. While the tester’s creativity plays a key role, they should anchor their approach in industry-recognized methodologies. Ask vendors about what drives their approach. For example, HBS derives its penetration methodologies from NIST SP800-115, the Open Web Application Security Project (OWASP), Open Source Security Testing Methodology Manual (OSSTMM), Penetration Testing Framework, and other industry best practices.

How Does the Vendor Help You Scope the Test?

A good vendor asks about your objectives. “We need to understand your perceived value of the test,” Jason Moulder says. “That helps us adjust the scope to either a more granular type of test or a broader test that incorporates all the elements required to address the scenario they have in mind.”

For example, telling a vendor you want to “do a pen test on our web app” could mean a lot of different things. Should the test be limited to the app itself? Should testers go after the infrastructure behind the app? If the tester can compromise the network via the app, should they keep going to see how much data they can access? Clear answers during scoping will produce the specific results you’re looking for.

Early on, you’ll make the key decision of deciding how much information to give the tester in advance. In a black box test, you tell them almost nothing about your environment. In gray box and white box tests, you give them different levels of information so that their work zeroes in on specific components.

How Will the Testers Incorporate Social Engineering?

Without a social engineering element, your pen test provides a very limited assessment of your security posture. Well-orchestrated, well-funded zero-day attacks grab a lot of headlines. But in the vast majority of cases, hackers rely on compromising end users in order to gain access to a system. So phishing tests, for example, should be part of a comprehensive pen test—and you should drill down on the vendor’s proposal there, too. “Generic phishing tests only provide about 60% of the potential value,” Jason says. “If you don’t test what can actually happen after someone clicks a fraudulent link, they don’t know actually know the impact it can have.”

How Useful is the Vendor's Final Report Format?

Ultimately, the pen test is only as valuable as the report it produces. This document tells you what the testers did, what they found and what they recommend for closing the gaps. Don’t pay for a glorified template full of boilerplate graphics that tell you little about your specific security posture. Ask for a sample report and review it carefully to determine whether it provides the kind of solid information you could act on.

A mediocre report, for example, may describe a weakness in your system. But a detailed report may show you that hackers would have to get through 10 other layers to exploit a weakness. With that information, you can decide whether the vulnerability is an acceptable risk for your organization.

Also, ask how the vendor plans to walk you through the report. You’re paying enough that you should expect analysis of the results, not just a PDF sent by e-mail. The consultant’s personal review often helps connect the data points into an overall picture. “Sometimes a thing by itself is no risk,” Jason says. “But if it’s chained with other things, it becomes a big risk.”

Does the Price Include a Retest?

The quote should include a retest of vulnerabilities at a set time (typically 90 days after the initial test). This gives your team time to address gaps and to get third-party validation that they were successfully remediated.

Does the Vendor Carry Liability Insurance?

Pen testing is invasive, and there’s a chance that the tester’s actions could cause performance interruptions in your system. Confirm that the vendor you’re considering has insurance to cover business interruptions, restoration costs, etc. (The fact that pen testers can seriously disrupt your operations should be another strong incentive to confirm that you’re hiring a true pro for this work.)

Clearly, many factors go into an effective pen test. That makes sense for a service that represents a significant investment in protecting your organization’s future. For help determining what your next pen test should entail, contact HBS today.