Case Study

The VLAN That Wasn't Really Segmented

> operator@pentest:~# for vlan in $(seq 1 12); do ping -c1 10.0.$vlan.1 && echo "VLAN $vlan: REACHABLE"; done<span class="cursor-blink">_</span>_

Peter Bassill 1 October 2024 16 min read
penetration-testing network-segmentation firewall-rules from-the-hacker-desk vlan-security lateral-movement compliance configuration-drift

Twelve VLANs on the diagram. One flat network in reality.

The network diagram was beautiful. It had been produced by a reputable network consultancy two years earlier as part of a redesign project. It showed twelve VLANs, colour-coded by function: user, server, management, voice, guest, DMZ, CCTV, building management, development, staging, PCI cardholder data environment, and a restricted VLAN for the board and senior leadership. Inter-VLAN routing was depicted with firewall symbols at every boundary, each annotated with a reference to a corresponding rule set document.

The diagram was laminated and pinned to the wall of the server room. It was referenced in the organisation's ISO 27001 documentation. It had been presented to external auditors during the most recent certification audit. It described a network architecture that was, on paper, well-segmented and properly controlled.

It bore almost no resemblance to the network we found when we plugged in.


The Engagement Brief

The client was a healthcare organisation — a private hospital group operating three sites, with a centralised IT function at the largest facility. They held ISO 27001 certification and were subject to NHS Data Security and Protection Toolkit requirements. Their network carried clinical systems, patient administration, medical imaging (PACS/DICOM), building management, and standard corporate IT.

We had been engaged to conduct an internal network penetration test with a specific focus on segmentation validation. The client's compliance team had identified network segmentation as a critical control in their risk treatment plan, and they wanted independent assurance that the segmentation was functioning as designed. The previous year's external audit had reviewed the network diagram and the firewall rule documentation and had found them satisfactory. Nobody had tested whether the rules actually worked.

We were given access points on three VLANs: the user VLAN, the guest VLAN, and the development VLAN. From each position, our objective was to determine what other VLANs were reachable, what services were accessible, and whether the firewall rules matched the documented policy.

The client expected us to confirm their segmentation. What we delivered was rather different.


The Documented Architecture

Before connecting to the network, we reviewed the documentation the client provided. The network design specified twelve VLANs with explicit inter-VLAN firewall rules governing permitted traffic flows.

VLAN Subnet Purpose Documented Access Policy
VLAN 10 10.0.10.0/24 Corporate Users DNS, Kerberos, SMB to servers; HTTP/S outbound; no access to clinical, PCI, or management
VLAN 20 10.0.20.0/24 Servers Accepts permitted traffic from users and clinical; no outbound initiation to user VLAN
VLAN 30 10.0.30.0/24 Management Restricted to IT admin workstations only; SSH, HTTPS, SNMP to infrastructure
VLAN 40 10.0.40.0/24 Clinical Systems HL7, DICOM to clinical servers; isolated from corporate users and guest
VLAN 50 10.0.50.0/24 PCI CDE Strictly isolated; only payment terminal traffic to acquiring bank
VLAN 60 10.0.60.0/24 Guest Wi-Fi Internet only; no access to any internal VLAN
VLAN 70 10.0.70.0/24 Voice / VoIP SIP and RTP to PBX only; isolated from data VLANs
VLAN 80 10.0.80.0/24 CCTV RTSP to NVR only; no external access; isolated
VLAN 90 10.0.90.0/24 Building Management BACnet to BMS controller; isolated from all other VLANs
VLAN 100 10.0.100.0/24 Development Access to dev servers only; no access to production, clinical, or PCI
VLAN 110 10.0.110.0/24 Staging Limited access to staging servers; no production access
VLAN 120 10.0.120.0/24 Executive / Board Restricted; dedicated file share and print; no cross-VLAN access from other user segments

On paper, this was a well-designed architecture. Clinical systems were isolated from corporate users. The PCI cardholder data environment was strictly segmented. Guest wireless had no internal access. The management VLAN was restricted to authorised administrators. Each boundary was governed by a documented firewall rule set.

We connected to the user VLAN and began testing whether reality matched the documentation.


Segmentation Testing — The Methodology

Segmentation validation requires systematic testing of connectivity between every VLAN pair. It is not sufficient to test a few representative paths — a single misconfigured rule can open a specific port between two specific VLANs whilst all other paths remain correctly blocked. The only way to validate segmentation is to test exhaustively.

From each of our three assessment positions (user, guest, and development VLANs), we tested connectivity to the gateway address of every other VLAN, followed by targeted port scans against known hosts on each segment. We tested common service ports (22, 23, 80, 135, 443, 445, 1433, 3306, 3389, 5432, 8080) plus a selection of OT and clinical protocols (502 for Modbus, 2575 for HL7 MLLP, 4242/11112 for DICOM).

The results were tabulated into a segmentation matrix — a grid showing the actual connectivity state between every VLAN pair, compared against the documented policy.


The Segmentation Matrix — Red Across the Board

We present the results from the user VLAN (10.0.10.0/24) as the primary assessment position, as it represents the most likely starting point for an internal attacker.

Segmentation Test Results — From User VLAN (10.0.10.0/24)
Target VLAN Documented Policy Actual Result
─────────────────────────────────────────────────────────────
VLAN 20 Servers DNS,Kerberos,SMB only ALL PORTS OPEN ✗ FAIL
VLAN 30 Management BLOCKED ALL PORTS OPEN ✗ FAIL
VLAN 40 Clinical BLOCKED ALL PORTS OPEN ✗ FAIL
VLAN 50 PCI CDE BLOCKED BLOCKED ✓ PASS
VLAN 60 Guest BLOCKED BLOCKED ✓ PASS
VLAN 70 Voice BLOCKED 22,80,443 OPEN ✗ FAIL
VLAN 80 CCTV BLOCKED 554,80 OPEN ✗ FAIL
VLAN 90 BMS BLOCKED ALL PORTS OPEN ✗ FAIL
VLAN 100 Development BLOCKED ALL PORTS OPEN ✗ FAIL
VLAN 110 Staging BLOCKED ALL PORTS OPEN ✗ FAIL
VLAN 120 Executive BLOCKED 445,3389 OPEN ✗ FAIL

Summary: 2 PASS / 9 FAIL — 82% segmentation failure rate

Nine of eleven VLAN boundaries failed segmentation testing from the user VLAN. Only the PCI cardholder data environment and the guest wireless network were properly isolated. Every other VLAN — including the management network, the clinical systems, the CCTV, the building management system, and the executive network — was reachable from a standard user workstation.

The results from the guest VLAN were better — the guest network was genuinely isolated from all internal VLANs except for one: the development VLAN was reachable on ports 80 and 443. This appeared to be a rule created to allow a development team member to test a web application from a mobile device on the guest network — a temporary exception that had become permanent.

The results from the development VLAN were catastrophic. Every VLAN except PCI and guest was reachable with no port restrictions whatsoever.

Critical Finding — Systemic Segmentation Failure

Eighty-two per cent of documented VLAN boundaries failed validation testing from the user VLAN. The network architecture documented in ISO 27001 evidence and presented to external auditors did not reflect the actual state of the firewall rule sets. Clinical systems, management infrastructure, CCTV, building management, and executive networks were all reachable from standard user endpoints.


Forensic Rule Analysis

With the cooperation of the client's network team, we conducted a forensic analysis of the firewall rule sets to understand how the segmentation had degraded so severely from the documented design. The firewall was an enterprise-grade appliance from a reputable vendor. It was capable of enforcing the documented policy. The problem was not the hardware — it was the configuration.

The firewall contained four hundred and forty-seven rules across its inter-VLAN policy. The original design had specified eighty-six rules. Three hundred and sixty-one rules had been added since the initial deployment — an increase of over four hundred per cent in two years.

Firewall Rule Set — Statistical Analysis
Firewall Rule Analysis — Inter-VLAN Policy

Total rules: 447
Original design rules: 86 (19%)
Rules added post-deployment: 361 (81%)

Rules with comments: 112 (25%)
Rules without comments: 335 (75%)

Rules referencing change tickets: 47 (11%)
Rules with no audit trail: 400 (89%)

Rules with 'any' as source: 68 (15%)
Rules with 'any' as destination: 54 (12%)
Rules with 'any' as service: 91 (20%)
Rules with 'any' in ALL fields: 23 ( 5%)

Rules with hit count = 0 (unused): 89 (20%)
Rules last hit > 12 months ago: 134 (30%)

Duplicate/overlapping rules: 37 ( 8%)
Rules shadowed by broader rules: 56 (13%)

The statistics painted a clear picture of configuration drift. Seventy-five per cent of rules had no comment explaining their purpose. Eighty-nine per cent had no associated change ticket. Twenty per cent of rules used 'any' as the service — permitting all ports between source and destination. Five per cent used 'any' in all three fields — source, destination, and service — effectively creating rules that permitted everything from everywhere to everywhere.

Twenty per cent of rules had never been hit — they permitted traffic that had never flowed, suggesting they were created speculatively or for testing and never removed. Thirty per cent had not been hit in over twelve months, suggesting they served applications or workflows that no longer existed.

Thirteen per cent of rules were shadowed — their intended restrictions were overridden by broader rules earlier in the policy that matched the same traffic with a more permissive action. These rules gave the appearance of control without providing it. Someone looking at the rule set would see a specific rule permitting only port 443 from the user VLAN to a particular clinical server. But higher in the policy, a broad 'any-any-any' rule — added during a troubleshooting session and never removed — matched the same traffic first, rendering the specific rule meaningless.


How Segmentation Dies

Through analysis of the rule comments, change tickets, and interviews with the network team, we reconstructed the timeline of how the segmentation had degraded. The pattern is remarkably common — we encounter variants of it on the majority of engagements where segmentation is in scope.

The Troubleshooting Exception
An application breaks. The network team is called. Under pressure to restore service, they add a temporary 'allow all' rule between the affected VLANs to eliminate the firewall as a variable. The application works. The rule is never removed. Over two years, twenty-three such rules had accumulated.
The Project Rule
A new system is deployed. The project team requests firewall rules for connectivity. The request specifies 'allow application traffic' without defining specific ports. The network team, unfamiliar with the application, creates a broad rule. The project completes. Nobody closes the rules.
The Cloned Rule
A new rule is needed for a similar use case to an existing one. Rather than creating a tailored rule, the existing rule is duplicated and modified — but the modification is incomplete, leaving a broader rule than intended. Over time, these clones accumulate, each slightly broader than the last.
The Knowledge Gap
The network engineer who designed the original architecture left the organisation. Their successor inherited a rule set they did not fully understand. Rather than risk breaking something by modifying existing rules, they added new rules on top. The policy grew without ever being reviewed.

The cumulative effect of these patterns is predictable and devastating. Each individual rule addition is small, justified, and often urgent. But over months and years, the specific, restrictive rules of the original design are buried under layers of broad exceptions. The segmentation on the diagram remains unchanged. The segmentation in the firewall has been eroded to the point of meaninglessness.

The organisation's change management process contributed to the problem. Firewall rule changes were classified as 'standard changes' — pre-approved modifications that did not require a full change advisory board review. This meant that individual rules could be added without independent scrutiny. The process required a change ticket, but enforcement was inconsistent — eighty-nine per cent of rules had no ticket reference.


Exploiting the Gaps

To demonstrate the impact of the segmentation failures, we conducted targeted exploitation from the user VLAN against systems that should have been unreachable.

Clinical Systems (VLAN 40)

The clinical VLAN hosted the hospital's Patient Administration System (PAS), electronic patient records, and a PACS (Picture Archiving and Communication System) server for medical imaging. From the user VLAN, we accessed the PACS server's DICOM interface on port 4242 — a protocol with no authentication by default. We were able to query the DICOM worklist and confirm the presence of patient imaging studies. We did not retrieve or view any images — our scope prohibited accessing patient data — but confirmed that the connectivity existed and that the DICOM service accepted queries from the user VLAN.

Management VLAN (VLAN 30)

The management VLAN contained the administrative interfaces for the core network switches, the firewall itself, the wireless controller, the VMware vCenter server, and the backup infrastructure. From the user VLAN, we accessed the vCenter web interface on HTTPS. The vCenter instance was running version 7.0 Update 2 — a version with known critical vulnerabilities. We authenticated to the vCenter using credentials recovered from a service account on the server VLAN (obtained through a separate finding). Full administrative access to the virtualisation platform — from a user workstation, across a boundary that the documentation stated was blocked.

Building Management (VLAN 90)

The BMS VLAN carried BACnet/IP traffic for HVAC control, environmental monitoring, and — in a hospital environment — critical systems including operating theatre air handling and pharmacy cold storage monitoring. From the user VLAN, we performed a BACnet Who-Is broadcast on port 47808. Sixteen BACnet devices responded. We could read temperature setpoints, air handling unit states, and alarm configurations. In a hospital, where environmental controls in operating theatres, intensive care units, and pharmaceutical storage have direct patient safety implications, this connectivity represented a risk that extended well beyond cybersecurity.

Cross-VLAN Access — Impact Demonstration
# Clinical VLAN — DICOM access from user VLAN:
$ findscu -v -S -k QueryRetrieveLevel=STUDY \
-aec PACS_SRV 10.0.40.20 4242
[RESPONSE] — 847 matching studies found

# Management VLAN — vCenter access from user VLAN:
$ curl -k https://10.0.30.10/ui/
[200 OK] VMware vSphere Client — login page served

# BMS VLAN — BACnet enumeration from user VLAN:
$ bacnet-scan --network 10.0.90.0/24
[RESPONSE] — 16 BACnet devices responding
AHU-OT-01 Operating Theatre 1 Air Handling Unit
AHU-OT-02 Operating Theatre 2 Air Handling Unit
FCU-ICU-01 ICU Fan Coil Unit
COLD-PHARM Pharmacy Cold Storage Monitor
...

From User VLAN to Everywhere

Step Action Weakness Exploited
01 Systematic connectivity testing from user VLAN to all 11 targets Nine of eleven VLAN boundaries failed — 82% failure rate
02 Accessed DICOM service on clinical PACS server Clinical VLAN reachable from user network; DICOM unauthenticated
03 Accessed vCenter management interface on management VLAN Management VLAN reachable; vCenter exposed to non-admin network
04 Enumerated 16 BACnet devices on BMS VLAN including theatre AHUs BMS VLAN reachable; BACnet unauthenticated by design
05 Accessed CCTV NVR feeds on CCTV VLAN via RTSP CCTV VLAN reachable from user network on port 554
06 Accessed executive file shares and RDP on executive VLAN Executive VLAN reachable on ports 445 and 3389
07 Forensic analysis of firewall — 447 rules, 361 post-deployment additions Configuration drift; no rule review process; 75% uncommented rules

Why Segmentation Fails

Network segmentation is the most frequently cited control in enterprise security architecture. It appears in every framework, every standard, and every best-practice guide. ISO 27001 references it. PCI DSS mandates it. NHS DSPT requires it. NIST recommends it. Every security consultant advocates for it.

And yet, on engagement after engagement, we find that segmentation exists in documentation but not in enforcement. The diagram on the server room wall describes a network that no longer exists — replaced over time by a tangle of exceptions, workarounds, and broad rules that have eroded every boundary the original design intended to enforce.

The reasons are consistent across organisations.

Rules Only Grow
Firewall rules are easy to add and terrifying to remove. Adding a rule is a standard change with immediate positive feedback — the application works. Removing a rule risks breaking something unknown. The result is a rule set that only ever grows, accumulating exceptions until the original policy is meaningless.
No Validation Testing
Segmentation is designed once and assumed to persist. Organisations verify that rules exist in the firewall configuration but do not verify that those rules produce the intended network behaviour. The gap between policy and enforcement widens invisibly.
Audit Theatre
External auditors review documentation — network diagrams, rule set exports, policy documents. They rarely test actual connectivity. An organisation can present a perfectly documented segmentation architecture that bears no resemblance to reality and pass an audit without challenge.
Operational Pressure
When a system breaks and users are affected, the pressure to restore service is immediate and intense. The pressure to maintain segmentation is abstract and deferred. In the moment, the temporary rule always wins. The temporary rule always becomes permanent.

Technique Mapping

T1046 — Network Service Discovery
Systematic port scanning across all VLAN boundaries to identify accessible services and validate segmentation enforcement.
T1021 — Remote Services
Access to management interfaces (vCenter, switch, firewall) and clinical systems (DICOM PACS) across VLAN boundaries that should have been blocked.
T1125 — Video Capture
CCTV NVR accessible via RTSP from the user VLAN, providing live and recorded surveillance feeds.
T0846 — Remote System Discovery (ICS)
BACnet enumeration of building management devices including operating theatre air handling units from the corporate user network.
T1133 — External Remote Services
RDP and SMB access to executive VLAN endpoints from the standard user network, bypassing the intended access restriction.

Recommendations and Hardening

Remediation Roadmap
Phase 1 — Immediate (0–14 days) Cost: Low
✓ Identify and remove all 'any-any-any' rules (23 rules)
✓ Block user VLAN access to management VLAN immediately
✓ Block user VLAN access to clinical VLAN immediately
✓ Block user VLAN access to BMS VLAN immediately
✓ Disable the guest-to-development rule (port 80/443)

Phase 2 — Short Term (14–90 days) Cost: Medium
○ Audit all 447 rules — remove unused (89) and stale (134) rules
○ Annotate every surviving rule with purpose and change ticket
○ De-shadow all overridden rules (56 rules)
○ Replace broad service rules with specific port/protocol rules
○ Rebuild inter-VLAN policy from documented design as baseline
○ Implement rule recertification process (quarterly review)

Phase 3 — Strategic (90–180 days) Cost: Medium–High
○ Implement automated segmentation validation (monthly testing)
○ Reclassify firewall changes from 'standard' to 'normal' change
○ Deploy firewall policy management tool with rule lifecycle tracking
○ Implement network traffic flow analysis (baseline normal traffic)
○ Establish rule expiry — all new rules require a review date
○ Include segmentation validation in annual pentest scope
○ Update ISO 27001 evidence with validated (not documented) state

The most impactful structural change is automated segmentation validation. A scheduled, automated test that verifies connectivity between VLAN pairs against the documented policy — run monthly or after every firewall change — would have detected the first deviation from the design within weeks. Several commercial and open-source tools exist for this purpose. The investment is modest; the alternative is discovering two years of accumulated drift during a penetration test.

Rule expiry is a deceptively simple control with transformative impact. Every new firewall rule should be created with a mandatory review date — typically ninety days for temporary rules and twelve months for permanent rules. When the review date arrives, the rule must be recertified by its owner with evidence that it is still required. If it is not recertified, it is disabled. This single policy change reverses the direction of drift: instead of rules accumulating indefinitely, they decay unless actively maintained.

The change management classification for firewall rules must be elevated. A firewall rule change is not a standard change — it is a modification to a security control that affects the organisation's risk posture. Every rule addition should require a defined scope (specific source, destination, and service — never 'any'), a business justification, a review date, and approval from a security-aware authority. This does not need to be bureaucratic — it needs to be consistent.

For healthcare environments specifically, access to clinical systems and building management from corporate VLANs must be treated as a patient safety issue, not merely a cybersecurity concern. An attacker with access to operating theatre air handling controls or pharmacy cold storage monitoring has the potential to cause physical harm. The segmentation boundaries around these systems must be tested with the same rigour applied to life-safety systems.


The diagram is not the network.

There is a well-known aphorism in philosophy: the map is not the territory. In network security, the equivalent is this: the diagram is not the network. A beautifully drawn, colour-coded, laminated network diagram tells you what someone intended the network to look like. It tells you nothing about what the network actually looks like today.

The only way to know whether segmentation works is to test it. Not to review the firewall rules — rules can be contradicted by other rules. Not to review the documentation — documentation reflects intent, not state. Not to rely on an auditor's assessment — auditors review evidence, not packet flows. The only reliable method is to stand on one side of the boundary and attempt to reach the other side. If you succeed, the boundary does not exist.

This organisation had invested in a proper segmentation design. They had paid a consultancy to produce it. They had deployed enterprise-grade equipment to enforce it. They had documented it for their ISO 27001 certification. And over two years, through the accumulated weight of four hundred individual decisions — each small, each justified, each urgent — they had dismantled it entirely.

Segmentation is not a project. It is not something you build once and walk away from. It is a living control that must be continuously monitored, validated, and defended against the relentless operational pressure to add just one more exception.

Until next time — stay sharp, stay curious, and test your segmentation. The diagram on your wall might be lying to you.

Legal Disclaimer

This article describes a penetration test conducted under formal engagement with full written authorisation from the client. All identifying details have been altered or omitted to preserve client confidentiality. No patient data was accessed or viewed during this assessment. The techniques described were performed within the scope of a legal agreement. Unauthorised access to computer systems is a criminal offence under the Computer Misuse Act 1990 and equivalent legislation worldwide. Do not attempt to replicate these techniques without proper authorisation.



If the answer is 'never', your diagram might be fiction.

Hedgehog Security conducts segmentation validation assessments that go beyond reviewing firewall rules. We test actual connectivity between every VLAN pair, compare results against documented policy, and deliver a segmentation matrix that shows you exactly where your boundaries hold and where they have failed. If your segmentation has not been validated since deployment, it is time.