Penetration testing or “pen testing” is an inherently manual process and runs in contrast to the DevSecOps movement. The DevSecOps movement is associated with outcomes such as automating everything, everything-as-code, CI/CD (continuous integration/continuous deployment) pipelines, and so on. When a pen test might take one to two weeks, where does it fit in? When should a team engage pen testers and how frequently? What should the new test triggers be? This article will explore these questions in more detail.
Fitting in a Pen Test
When a pen test and software release process are tightly coupled, the pen test inevitably slows it down. Rolling deployments are paradoxical with pausing everything for a pen test to occur.
In my experience, it’s useful to think about scheduling in a couple of ways:
- Pause and test before significant releases into production — i.e. the initial product launch, major re-writes (if that is happening), or seasonal code freezes, such as holidays.
- Allocating test environments that receive rolling deployments to have pen testing that occurs out-of-band.
New Test Triggers
This comes up frequently in regulated environments where it’s assumed by some that a certain amount of change amounts to a controls re-assessment. In that controls process re-assessment, a pen test or some other kind of security evaluation (or both) may be “required” for the team. This, in part, becomes a matter of interpretation, likely mapped back to a change control measure, ensuring that the changes being introduced to an environment don’t introduce some unknown or unacceptable level of risk to the assets involved.
Test Frequency
The higher risk of the application, the more frequently it should go through analysis. But that’s not a blanket rule; blanket rules are often not helpful in security and can be outright counterproductive. If the application is something that isn’t going through significant changes very frequently, but is very high risk, then a pen test would likely yield a low return on investment. If an application is backed up with good test coverage and complementary security activities such as dependency scanning, proactive threat modeling, static code analysis, and perhaps has an in-line web application firewall, then more frequent pen tests may yield a low return on investment.
Return on Investment
In the last section, I used the term return on investment (ROI) as part of the potential pen test decision criteria. Unless there is some verifiable regulatory requirement for conducting a particular security activity on a particular schedule, then the organization should be looking at the ROI of any given security activity. Just because pen testing is done quarterly works in one organization doesn’t mean the same results will translate in another.
What does return on investment mean in the context of pen testing? We can look at this in a few different ways:
- Number and quality of results based on the cost of each test
- Diversity of results from pen testing compared to other security activities such as static code analysis, threat modeling, or lighter-weight security-focused lifting (e.g., solutions like Semgrep)
- Results identified per the number of resources invested to conduct the test (e.g., length of time spent testing, number of testers, etc.)
- Coverage received from testing
- Diversity of results from each test iteration. For example, if you’re doing tests quarterly yet finding 80% of the same issues or the same classes of issues then it’s likely time to re-invest your resources into remediation efforts or explore how your team could support a systematic prevention of a class of vulnerabilities that continues to affect you.
ROI may mean something totally different to you and your organization, but that’s okay. The above points are things that I’ve personally found useful in my own decision-making process over the years. The point is, that you’re being intentional with your decisions and the way you allocate resources. Just because something was once a good idea doesn’t mean it’s still a good idea and just because something works in another organization, doesn’t mean it will work in yours.
Want more cybersecurity insights? Visit the Cybersecurity channel:
