> scope --validate /engagements/latest.yml && echo 'FAIL'_
This sounds counterintuitive. Surely any testing is better than no testing? A partial view is better than blindness? Not necessarily — and understanding why is critical for anyone who commissions, approves, or relies on penetration test results.
An organisation that has never been tested knows it hasn't been tested. That knowledge — uncomfortable as it is — creates a healthy caution. Decisions about risk carry an implicit caveat: we don't know what an attacker would find. Budget discussions acknowledge that there's unquantified exposure. The absence of data is, at least, honest.
An organisation that has been tested with a badly-defined scope believes it has been tested. The report sits in the shared drive. The executive summary says "no critical findings." The board receives assurance that the security posture is acceptable. Decisions about risk are made with confidence — confidence that is entirely misplaced, because the test didn't assess the things that actually matter.
This is the danger. A badly-scoped pen test doesn't just fail to find vulnerabilities. It generates false evidence that they don't exist. And false evidence is worse than no evidence, because it leads to action — or more precisely, to inaction where action is urgently needed.
Nobody makes bad decisions because they know they have no data. People make bad decisions because they believe they have good data — and the data is wrong. A badly-scoped pen test is wrong data dressed in a credible report.
Bad scoping rarely looks bad at the time. It usually looks efficient, cost-conscious, and pragmatic. The problems only become apparent later — when a breach exploits exactly the layer that wasn't tested, or when an auditor asks why the pen test covered 12 hosts but the asset inventory lists 200.
Here are the patterns we encounter most frequently — each one leading to a test that looks complete on paper but misses the mark in practice.
| Scoping Failure | How It Looks | What Actually Happens |
|---|---|---|
| Budget-led scoping | "We have £5,000 — just test whatever fits within that." The provider reverse-engineers a scope to match the budget rather than the risk. | The test covers a fraction of the attack surface — typically the easiest, lowest-risk targets. Critical systems are excluded because they'd push the cost up. The report says "tested" but the business-critical assets were never touched. |
| Compliance-minimum scoping | "Just test the cardholder data environment — that's all PCI requires." The scope is drawn at the exact boundary of the compliance requirement and nothing more. | The CDE passes. The attacker enters through the corporate network — which was out of scope — and pivots into the CDE from the inside. The compliance test was technically correct but operationally useless. |
| Known-asset-only scoping | The client provides a list of systems they know about. The tester tests that list. Nobody discovers the forgotten staging server, the shadow IT cloud account, or the legacy subnet nobody documented. | The test covers the managed estate. The attacker targets the unmanaged estate — the systems that exist precisely because nobody's paying attention to them. |
| Technology-only scoping | "Test our servers and web application." The scope covers technology but excludes people, processes, and physical security entirely. | Social engineering — the most common initial access vector in real attacks — is untested. The phishing email that would compromise 30% of staff goes undetected. The helpdesk that would reset a password over the phone without verification is never challenged. |
| Copy-paste scoping | The same scope from last year's test is reused without review. No account is taken of new systems, decommissioned servers, cloud migrations, or application changes. | The test assesses last year's environment. This year's attack surface — new applications, expanded cloud estate, additional office locations — is untested. The test provides assurance about an environment that no longer exists. |
| Depth-avoidance scoping | A broad scope covering many systems, each tested shallowly. Vulnerability scanning masquerading as penetration testing. Every host gets a port scan; none get manual exploitation. | The report lists 200 findings sorted by CVSS score. None have been verified. None have been chained. The critical business logic flaw in the customer portal — the one that actually matters — isn't in the report because no human looked at it. |
False confidence from a badly-scoped test doesn't just sit in a report. It propagates through the organisation, influencing decisions at every level — from the security team's remediation priorities to the board's risk appetite.
These are composite scenarios drawn from real engagements — anonymised but not hypothetical. Each one illustrates how a scoping failure led to a false conclusion.
| The Situation | The Scoping Failure | The Outcome |
|---|---|---|
| A logistics company commissioned an annual external pen test covering their 20 public IPs. | Nobody updated the scope when 30 new cloud-hosted services were deployed during the year. The cloud estate had no firewall, no WAF, and default credentials on three admin panels. | The pen test report was clean. Four months later, an attacker found one of the cloud admin panels via Shodan, logged in with default credentials, and accessed the production database containing 80,000 customer records. |
| A SaaS provider tested their customer-facing web application annually. Grey-box, OWASP-aligned, thorough. | The internal API that powered the application was never in scope. It was considered "internal" even though the web application made client-side calls to it. The API had no rate limiting and returned full user objects including hashed passwords. | A researcher discovered the API, enumerated 12,000 user accounts, and responsibly disclosed. The SaaS provider's response: "but we pen test every year." They did — they just tested the skin, not the skeleton. |
| A law firm commissioned a pen test to satisfy a client contractual requirement. The client specified "annual penetration testing of internet-facing systems." | The firm scoped the cheapest possible test: external infrastructure only, no web application testing, no social engineering. Their internet-facing systems included a client portal with document upload functionality. | The portal had a file upload vulnerability that allowed server-side code execution. An attacker uploaded a web shell, accessed the document store, and exfiltrated privileged client documents. The contract required testing of internet-facing systems — the portal was internet-facing — but the scoping chose infrastructure over application. |
| A retail chain commissioned an internal infrastructure test across their head office network. | Branch office networks were excluded to reduce cost. All 45 branches connected to the head office via site-to-site VPN with no segmentation. A compromised branch device had a direct route to the head office domain controller. | An attacker compromised a poorly-secured EPOS system in a branch, traversed the VPN, and reached the head office AD. The internal pen test at head office found no critical issues — because the attack path started 200 miles away. |
Scoping failures aren't random. They stem from a small number of recurring root causes — most of which are organisational rather than technical.
| Root Cause | How It Manifests |
|---|---|
| No asset inventory | You can't scope what you don't know exists. Without a current, comprehensive asset inventory, the scope is always a best guess — and attackers don't limit themselves to your best guess. |
| Procurement owns the process | When the pen test is procured like a commodity — three quotes, cheapest wins — the scope is optimised for price rather than coverage. The provider who cuts the most corners gives the lowest quote. |
| No threat model | Without understanding who would attack you, why, and how, you can't scope a test that simulates realistic threats. The scope defaults to "test everything we can afford" rather than "test what an attacker would target." |
| Scope is set by IT alone | IT knows the technology. They rarely know which systems hold the most commercially sensitive data, which processes would cause the most business disruption if compromised, or which clients have contractual security expectations. Scoping needs business input. |
| Last year's scope is reused | Environments change continuously — new applications, cloud migrations, acquisitions, new office locations. Reusing last year's scope tests last year's environment. The delta is where the risk lives. |
| The provider doesn't push back | A good provider challenges a bad scope. A provider who takes your scope at face value without questioning whether it reflects your actual risk is either inexperienced or indifferent. Both are dangerous. |
After scoping is complete, ask yourself: if we suffered a breach tomorrow, would this test have covered the systems and attack paths involved? If the answer is "probably not" or "we don't know," the scope needs rethinking.
Good scoping isn't complicated, but it requires discipline and the right conversations. Here's a practical framework that consistently leads to better-scoped, higher-value engagements.
| Principle | What It Means in Practice |
|---|---|
| Start with risk, not with budget | Identify your crown jewels first: customer data, financial systems, intellectual property, operational infrastructure. Scope the test around the attack paths to those assets. Then discuss budget — and if the budget doesn't cover the risk, that's a conversation worth having. |
| Involve the business | Invite someone from outside IT to the scoping conversation. The CFO knows which data would trigger a regulatory notification. The operations director knows which systems would halt the business if compromised. Their input shapes a scope that reflects business risk, not just technical inventory. |
| Discover before you scope | Run a basic asset discovery exercise before defining the scope. Enumerate your external attack surface, review your cloud accounts, and check your asset inventory against reality. You cannot test what you don't know about — and you will be surprised by what you find. |
| Favour depth over breadth | If budget forces a trade-off, test fewer things more thoroughly rather than many things superficially. A deep test of one critical application will find the business logic flaws and chained attack paths. A shallow scan of 50 hosts will find nothing a vulnerability scanner wouldn't. |
| Document what's excluded — and why | Every exclusion should be a conscious decision with a documented rationale. "Excluded because it's out of budget" is honest. "Excluded because we forgot it existed" is a gap. Writing exclusions down forces the conversation about whether they're acceptable. |
| Review the scope annually — from scratch | Don't roll forward last year's scope. Rebuild it each year starting from current business risk, current asset inventory, and current threat intelligence. Treat every engagement as if it's your first. |
| Choose a provider who challenges you | The best scoping conversations involve pushback. A provider who asks "why is the API excluded?" or "what about your branch networks?" or "have you considered social engineering?" is adding value. A provider who silently accepts whatever you give them is not. |
The quality of the scoping conversation is a reliable proxy for the quality of the test. Here are the questions you should expect a good provider to ask — and the red flags that suggest they're not invested in getting the scope right.
| A Good Provider Asks... | A Red Flag Provider Asks... |
|---|---|
| "What's the objective — what do you want to learn from this test?" | "How many IPs do you have?" |
| "What are the most business-critical systems and data in your environment?" | "Can you send us a list of servers?" |
| "Has anything changed since the last test — new systems, cloud migration, acquisitions?" | "Shall we just use the same scope as last year?" |
| "Have you considered testing the human layer — phishing, vishing, physical?" | (Doesn't mention social engineering at all) |
| "What's excluded and why? Are you comfortable with those exclusions?" | (Accepts the scope without questioning exclusions) |
| "Who's the audience for the report — the security team, the board, an auditor, a client?" | "You'll get the standard report format." |
| "What happens if we find something critical — who do we call and how fast?" | (No discussion of critical finding procedures) |
| "Is this budget sufficient for the scope you need? If not, let's prioritise." | "We can fit that in the budget" (without explaining what's being cut) |
Every pen test involves a trade-off between breadth (how much is tested) and depth (how thoroughly each element is tested). Budget and time are finite. Understanding this trade-off — and making it consciously — is what separates useful testing from security theatre.
| Broad + Shallow | Narrow + Deep | |
|---|---|---|
| What it covers | Many systems, each assessed quickly — port scans, service enumeration, automated vulnerability checks | Fewer systems, each assessed thoroughly — manual testing, exploitation, chained attack paths, business logic |
| What it finds | Known CVEs, default credentials, missing patches, SSL/TLS misconfigurations — the same things a vulnerability scanner finds | Business logic flaws, authorisation bypasses, chained attack paths, privilege escalation, real-world impact demonstrations |
| False positive rate | High — automated findings without manual verification | Low — every finding is manually confirmed and demonstrated |
| Business context | Minimal — findings listed by CVSS score with no contextual prioritisation | Detailed — findings mapped to business impact, with attack narratives explaining the real-world consequences |
| What it misses | Everything that requires manual investigation, creative thinking, or time — which is where the critical findings live | Systems outside the narrowed scope — which may have their own vulnerabilities |
| Best used when | You need a broad baseline across a large estate and will follow up with deeper testing of high-risk areas | You know which systems carry the most risk and want thorough assurance of those specific assets |
Neither approach is wrong in isolation. The mistake is choosing broad-and-shallow when what you need is narrow-and-deep — or worse, believing that broad-and-shallow constitutes a genuine penetration test. It doesn't. It's a vulnerability assessment with a misleading title.
Sometimes scoping failures only become apparent when you read the report. Here are the signs that the scope may have been wrong — and that the results should be treated with caution rather than confidence.
| Warning Sign | What It Suggests |
|---|---|
| The report has no critical or high findings | This might mean you're genuinely well-secured. It might also mean the scope avoided the high-risk areas. Ask: were the crown jewels in scope? Was the internal network tested? Was the application tested beyond automated scanning? |
| Every finding is an automated scanner result | If the report reads like Nessus output with a logo on the front, the test lacked manual depth. Manual testing is where the critical findings — business logic, chained attacks, privilege escalation — are discovered. |
| The scope section is vague or absent | If the report doesn't clearly state what was tested, what was excluded, and what approach was used, you can't assess whether the findings (or lack thereof) are meaningful. |
| The scope hasn't changed in three years | Your environment has changed. If the scope hasn't kept pace, the test is assessing a historical version of your organisation. |
| The tester didn't ask questions during scoping | A provider who doesn't ask about your business, your threats, or your concerns isn't scoping to your risk — they're scoping to your budget. |
A badly-scoped penetration test is worse than no test at all because it replaces honest uncertainty with false confidence. It tells the board you've been assessed when you haven't been — at least not in the places that matter. It diverts remediation budget to low-risk findings while critical vulnerabilities go undiscovered. It satisfies a checkbox while leaving the organisation exposed.
The fix isn't more testing — it's better scoping. Start with risk, not budget. Involve the business, not just IT. Discover your assets before you define your scope. Favour depth over breadth. Document your exclusions. Review the scope from scratch every year. And choose a provider who pushes back when the scope doesn't match the risk.
The most valuable sentence in a pen test report isn't a critical finding. It's a clear, honest, exhaustive description of what was tested and what was not — because that's what allows everyone reading the report to understand exactly how much confidence the results deserve.
We invest more time in scoping than most providers — because we've seen what happens when it's rushed. Every engagement starts with a detailed conversation about your risk, your environment, and your objectives.