Lumen Technologies’ Black Lotus Labs finally reveals what happened to the US ISP last October
An investigation by Lumen Technologies’ Black Lotus Labs has revealed what happened when more than 600,000 small office/home office (SOHO) routers were taken offline, with all belonging to a single ISP. The incident took place over a 72-hour period between October 25-27, rendered the infected devices permanently inoperable, and required a hardware-based replacement.
Public scan data confirmed the sudden and precipitous removal of 49% of all modems from the impacted ISP’s autonomous system number (ASN) during this time period. While Lumen demurred to name the ISP, Reuters has linked the massive outage to Windstream after customers began flooding message boards last October with reports their routers had suddenly stopped working and remained unresponsive to reboots and all other attempts to revive them.
The October incident, which was not disclosed at the time, was one of the most serious cyberattacks ever against the US’s telecommunications sector.
Lumen’s analysis identified “Chalubo,” a commodity remote access trojan (RAT), as the primary payload responsible for the event. This trojan, first identified in 2018, employed savvy tradecraft to obfuscate its activity; it removed all files from disk to run in-memory, assumed a random process name already present on the device, and encrypted all communications with the command and control (C2) server.
“We suspect these factors contributed to there being only one report on the Chalubo malware family to date,” wrote the authors in a blog post. “Chalubo has payloads designed for all major SOHO/IoT kernels, pre-built functionality to perform DDoS attacks, and can execute any Lua script sent to the bot. We suspect the Lua functionality was likely employed by the malicious actor to retrieve the destructive payload.”
Lumen’s global telemetry indicates the Chalubo malware family was highly active in November 2023 and remained so into early 2024. Based on a 30-day snapshot in October, Lumen identified over 330,000 unique IP addresses that communicated with one of 75 observed C2 nodes for at least two days, indicating a confirmed infection.
This suggests that while the Chalubo malware was used in this destructive attack, it was not written specifically for destructive actions. “We suspect the threat actors behind this event chose a commodity malware family to obfuscate attribution, instead of using a custom-developed toolkit. At this time, we do not have an overlap between this activity and any known nation-state activity clusters,” said Lumen. “We assess with high confidence that the malicious firmware update was a deliberate act intended to cause an outage, and though we expected to see a number of routers make and models affected across the internet, this event was confined to the single ASN.”
Attacking two router vendor models
The attack notably affected rural and underserved communities within the ISP’s service area. Residents in these areas potentially lost access to emergency services, agricultural monitoring systems, and healthcare providers, highlighting the severe consequences of the disruption.
Lumen saw that initial complaints emerged in late October 2023, focusing on ActionTec T3200s and T3260s devices, which showed a static red light. Investigation revealed a firmware issue leading to a significant drop in the number of exposed devices, confirmed by scan data from Censys. The ISP’s ActionTec and Sagemcom F5380 modems were primarily affected. Lumen observed a drop of around 179k IP addresses that had an ActionTec banner and a drop of around 480k devices associated with Sagemcom.
Ramifications of the attack
Black Lotus Labs said the investigation stood out for two reasons. First, this campaign resulted in a hardware-based replacement of the affected devices, which likely indicates that the attacker corrupted the firmware on specific models. “The event was unprecedented due to the number of units affected – no attack that we can recall has required the replacement of over 600,000 devices.”
In addition, they said this type of attack has only ever happened once before, with AcidRain used as a precursor to an active military invasion. “At this time, we do not assess this to be the work of a nation-state or state-sponsored entity,” said Lumen. “In fact, we have not observed any overlap with known destructive activity clusters; particularly those prone to destructive events such as Volt Typhoon, or SeaShell Blizzard.”
The second unique aspect is that this campaign was confined to a particular ASN. “Most previous campaigns we’ve seen target a specific router model or common vulnerability and have effects across multiple providers’ networks,” said Lumen.
“Destructive attacks of this nature are highly concerning, especially so in this case,” concluded Lumen. “A sizeable portion of this ISP’s service area covers rural or underserved communities; places where residents may have lost access to emergency services, farming concerns may have lost critical information from remote monitoring of crops during the harvest, and health care providers cut off from telehealth or patients’ records.”