A Medical Practice in Plano, TX Had a Device Beaconing to China for 17 Hours Through a Channel the Firewall Could Not See
A HIPAA-covered multi-provider practice had 66 firewall rules, 26 alert rules, and an unusable security mailbox. They could not answer whether any of it was working. We deployed a syslog collector, analyzed 2.7 million log lines in 24 hours, and found out.
The Challenge
This practice had invested in a capable next-generation firewall. On paper the configuration looked thorough: 66 firewall rules, 26 alert rules, and 6 automation triggers. The problem was that nobody could verify whether any of it was actually functioning as intended. Alert email had become unreadable. Staff could not say which alerts mattered, which were noise, and which had silently stopped firing altogether.
As a HIPAA-covered entity handling protected health information across a single location with roughly 100 endpoints and 20 IP cameras, the practice had a legal obligation to maintain an accurate, functioning security program. The gap between what the firewall configuration said it was doing and what it was actually doing was unknown and unquantifiable.
They needed to know whether their controls were real. That required getting the logs, reading them in full, and testing every rule assumption against actual traffic.
Alert Fatigue
Security email was completely unusable. One misconfigured rule had rendered the alert channel unreliable for the entire team.
Unverifiable Rules
No one could confirm whether detection rules were firing, matched the right traffic, or had ever triggered in production.
HIPAA Obligations
As a covered entity, the practice was required to maintain documented, functional technical safeguards for PHI. The existing configuration could not satisfy that requirement.
What We Found
We deployed a syslog collector and captured 24 hours of full-fidelity logs: 2.7 million log lines, 690,955 sessions. Within the first four hours, we had identified three systemic detection failures and one active compromise.
21 Attack-Detection Rules Had Never Fired
The firewall had 21 inbound rules labeled as "log attack attempts." None of them had ever generated a single match. The reason was architectural: traffic to closed ports is dropped by the stateful packet filter before it reaches the firewall's application layer, so the rules waiting at the application layer never had traffic to evaluate. The practice believed these rules were actively recording inbound attack attempts. They were not.
In 24 hours of log data, 1,062 real attack sessions went completely unlogged: 540 Telnet probes, 218 SSH attempts, 63 RDP attempts, and additional database-port probes. These sessions were dropped at the packet layer but not recorded anywhere the team could see. The attack surface existed; the visibility did not.
Every "Data Exfiltration" Alert Was Mislabeled
The firewall had 20 outbound alert rules intended to detect insider data exfiltration. The rules matched on destination interface alone. Because destination-interface logic in this firewall matches both outbound and inbound traffic, every one of those 20 rules was triggering on inbound attack scans, not outbound data movement.
All 137 sessions matching the "data exfiltration" alert category were inbound scans. The practice had been generating insider-threat alerts that represented zero insider activity and zero data leaving the network. No actual exfiltration detection was in place.
137 sessions flagged as data exfiltration. All 137 were inbound scans. Zero were outbound data movement.
One Rule Generated 98% of All Alert Volume
A single misconfigured alert rule produced 220,890 events in 24 hours. Email notification was enabled on that rule with no rate limit. This was the direct cause of the unusable security mailbox: one rule, generating over 220,000 email-eligible events per day, had buried every other alert the system generated.
That one rule represented 98% of all alert volume across the entire environment. The remaining 2% of the alert stream, which included actual signals, was invisible in the noise.
A Wireless Device Had Been Tunneling Data to Chinese Infrastructure for 17 Hours
At hour 8, log analysis identified a wireless device on the network that had been beaconing to Chinese IP infrastructure for at least 17 hours. The device was a USB-over-IP adapter manufactured in Shenzhen, running 2020 firmware. It had been placed in quarantine by the firewall, but the quarantine was only partially effective.
Quarantine Traffic Breakdown
| Traffic Type | Sessions | Data Out | Result |
|---|---|---|---|
| Inspected sessions (application layer) | 36,231 | 0 bytes | Blocked (100%) |
| ICMP sessions (transit, bypassed quarantine) | 5,171 | 408 KB | Not Blocked |
The firewall's quarantine policy blocked all inspected sessions: 36,231 sessions, zero bytes exfiltrated. But transit ICMP traffic never reaches the firewall's application layer. The quarantine rules, written at the application layer, could not see it. The device exploited this gap to run an ICMP tunnel, sending 408 KB to a Chinese DNS provider over 17 hours through 5,171 ICMP sessions that bypassed the quarantine entirely.
The traffic had a textbook ICMP tunnel signature: uniform payload sizes of approximately 79 bytes per packet, near-zero return traffic, a fixed foreign destination, and a consistent 17-hour transmission window. A normal ICMP ping produces variable payload sizes and bidirectional traffic. None of that was present here.
Containment was applied at layer 2: a DHCP deny and a wireless controller block. The device was removed from the network entirely. Beaconing activity stopped within 2 minutes.
Additional Findings
19 Hostile Scanner Networks Blocked
Log analysis identified 19 hostile scanner networks in the first 24 hours. Among them was a coordinated Bulgarian /24 sweep using 15 source IPs across 547 distinct ports. Each IP in the sweep covered non-overlapping port ranges, a deliberate technique to evade per-source-IP rate limits while completing a full port map collectively. The coordinated nature of the sweep was only visible by correlating across all 15 source addresses simultaneously. The combined sweep produced 324 blocked hostile sessions in the first 5 hours.
False Positive Corrected: Security Camera Quarantined for Uploading Video
One of the 20 IP cameras on the network had been placed in quarantine. The triggering alert was "219 MB in 18 minutes to AWS S3," categorized as a data exfiltration event. Review of the log data showed that all 19 other cameras were doing exactly the same thing: uploading video to cloud storage. The quarantined camera was not behaving anomalously. It had been recording nothing for months because quarantine blocked its upload sessions entirely.
19,661 blocked sessions over months. Zero MB of footage uploaded. The camera was quarantined for normal behavior. It was releasing zero footage to the monitoring system during that time.
What We Rebuilt
After validating each rule against actual traffic logs, we rebuilt the firewall and alert rule sets from the ground up and deployed an IPS configuration tuned to the practice's traffic baseline. Work was completed within 9 hours of initial log collection.
Firewall Rule Rebuild
Expanded from 66 to 74 rules. Corrected 21 existing rules that had never fired or were misattributed. All rules validated against 24-hour log traffic before deployment. See our managed SOC service for ongoing firewall management details.
Alert Rule Rebuild
Expanded from 26 to 42 alert rules. Corrected 4 rules including the one generating 220,890 events per day. Rate limiting applied. Noise suppression achieved 98% reduction in alert volume on day one of deployment.
IPS Deployment and Tuning
Intrusion Prevention System enabled and tuned against the practice's traffic baseline. First-day noise suppressed to 2% of pre-deployment volume. IPS now provides application-layer inspection coverage for threat categories the previous rule set did not reach.
Full-Fidelity Log Retention
Syslog collector deployed and maintained. All session and event data is stored with searchable history. The practice now has an auditable log record that satisfies HIPAA audit control requirements. See our compliance services for HIPAA documentation requirements.
Compromise Containment
The beaconing device was removed from the network via DHCP deny and wireless controller block. Layer-2 containment was selected because the firewall's application-layer quarantine had already proven insufficient for transit ICMP. Activity stopped within 2 minutes of containment application.
Results
All numbers come from client log data.
690,955 sessions reviewed in 24 hours of capture. Provided the first complete, verified picture of what the network was actually doing.
Rules that had never fired in production were corrected and validated. Attack attempts that had been silently dropped without logging are now recorded.
Daily alert volume from one misconfigured rule, eliminated on day one. The security mailbox became readable for the first time since the rule was created.
From identification of the ICMP tunnel to full device removal from the network. The compromise had been active for 17 hours before log analysis surfaced it.
Timeline
From syslog deployment to active compromise containment in under 9 hours.
Syslog Deployed, 24-Hour History Captured
Syslog collector deployed to the network. 24 hours of firewall log history pulled and ingested. 2.7 million log lines, 690,955 sessions available for analysis.
Rule Attribution Validated, Three Systemic Failures Confirmed
Every firewall and alert rule cross-referenced against actual log traffic. The 21 never-fired rules, the mislabeled exfiltration alerts, and the 220,890-event noise rule all identified and documented.
Corrected Rule Set Deployed, Hostile Scanner Blocks Live
Rebuilt firewall rule set (66 to 74 rules) and alert rule set (26 to 42 rules) deployed. 19 hostile scanner networks blocked. Coordinated Bulgarian /24 sweep producing 324 sessions blocked within the first 5 hours of new rule deployment.
Active Compromise Identified and Contained
ICMP tunnel to Chinese infrastructure identified. Device isolated via DHCP deny and wireless controller block. Beaconing activity stopped within 2 minutes. 408 KB confirmed sent via tunnel over 17 hours prior to containment.
IPS Deployed and Tuned
Intrusion Prevention System enabled and tuned against the practice's traffic baseline. 98% of first-day noise suppressed. Alert channel became readable.
Network Segmentation and Detection Tuning in Progress
Camera network segmentation, endpoint segmentation, and continued IPS tuning underway. See open items below.
Open Items
Not everything is resolved. These items are in progress.
Camera Network Segmentation
The 20 IP cameras currently share network segments with clinical endpoints. Segmenting camera traffic onto an isolated VLAN is in progress. Until complete, a compromised camera has lateral movement potential toward clinical systems. The false-positive quarantine incident demonstrated the need for purpose-built camera network policies.
Licensing Gap
The full IPS and threat intelligence feed capabilities require a licensing tier the practice does not currently hold. Some advanced detection categories are available but not fully activated. A licensing upgrade is under review. Current IPS coverage is functional but not at full capability.
Services Applied in This Engagement
This engagement drew on firewall analysis, log review, threat detection, and compliance documentation capabilities.
Managed SOC
Log collection, rule validation, alert management, and ongoing threat monitoring.
Learn morePenetration Testing
Validating security controls through adversarial testing to find what scanning tools miss.
Learn moreHIPAA Compliance
Audit-ready documentation, risk analysis, and technical safeguard implementation for covered entities.
Learn moreDo You Know What Your Firewall Is Actually Doing?
A firewall configuration that looks correct on paper and one that is working correctly are not the same thing. Log analysis tells you which one you have.
We serve medical practices, clinics, and healthcare organizations across the Plano and DFW area. Start with a free consultation.