Despite these initial indicators, the tremendous scope of the espionage campaign and its sophistication only became clear last week, after the elite cybersecurity firm FireEye disclosed a devastating data breach on its own network.
The US government’s early detection, which has not been previously reported, did not provide conclusive evidence that the government’s networks had been compromised, but it was enough to worry top cybersecurity officials that potential vulnerabilities existed.
The revelation illustrates how a select few within the government’s most classified corners grappled with early warning signs of the massive hack — and launched into a months-long investigation that ended up uncovering links to the devastatingly sophisticated spying operation that has rocked Washington this week.
At least a half a dozen federal agencies are now known to have been targeted, including the Department of Homeland Security’s cyber arm and the Departments of Agriculture, Commerce, Energy and State.
Investigators are still trying to determine what, if any, government data may have been accessed or stolen in the hack. The indicators identified during early detection efforts did not reveal evidence of a classified data breach, two sources told CNN.
Two sources described the suspicious activity detected months ago as a “backdoor-enabled persistent threat” consistent with the ongoing hacking effort disclosed this week, and added there is still no indication the hackers accessed classified systems or information.
At the time, officials probing the activity were unable to tie it to the specific IT management software that has been identified as a source of infection in other agencies.
The National Security Agency did not respond to CNN’s request for comment. US CyberCommand declined to comment.
Much of the federal government only learned of one of the country’s worst-ever cybersecurity incidents from public reporting and disclosures from private firms.
On December 8, FireEye disclosed that it had been the target of a sophisticated, likely state-sponsored espionage attempt, and that several of its own hacking tools had been stolen.
Then, on December 13, Reuters first reported that the Departments of Commerce and Treasury had been hit by hackers. The Commerce Department soon confirmed a security incident.
That same evening, FireEye said it had identified the source of its own intrusion: Malware hidden in the legitimate software updates published by a widely used IT management firm known as SolarWinds.
The updates containing the malware were distributed to as many as 18,000 SolarWinds customers, including US government agencies and Fortune 500 companies. The announcement touched off a mad scramble by federal agencies to determine if the infected software had been installed on their networks.
The Department of Homeland Security’s cyber agency, the Cybersecurity and Infrastructure Security Agency, issued an emergency directive — only the fifth in its five-year history — instructing all federal agencies to review their systems and to shut down any affected SolarWinds installations. CISA didn’t immediately respond to a request for comment.
CISA quickly became a central figure in the US government’s response, holding multiple conference calls this week with federal, state and local officials as well as private sector leaders, according to Daniel Dister, the chief information security officer for the state of New Hampshire, who participated in the calls.
CISA has gamely provided what information it can to a vast array of audiences hungry for answers, Dister said. But other security experts say that what the public is demanding of CISA far exceeds the support it has been given.
“They’re supposed to be the federal agency to help the federal government with cybersecurity,” said Robert Lee, CEO of the cybersecurity firm Dragos. “But what they’ve done, and what Congress has asked them to do, is partner with industry, offer services and free penetration tests. That was never something they were set up or structured to do, and never something they were resourced to do broadly.”
The burden on CISA to investigate the hack is only likely to grow as evidence mounts of a multi-pronged penetration campaign by the suspected Russian hackers.
“SolarWinds was not the only path. It would be strange for any actor of this capability to rely on any single method of entry,” John Hultquist, FireEye’s senior director of intelligence analysis, told CNN.
CISA warned on Thursday that it had found evidence of other forms of compromise, but declined to elaborate other than by citing an outside security firm’s research.
This story has been updated with additional information.
CORRECTION: An earlier version of this story misstated John Hultquist’s title at FireEye. He is the senior director of intelligence analysis.
Jeremy Herb, Jim Sciutto and Alex Marquardt contributed.