David Bridges
“On September 5th, Twitter experienced the loss of its Sacramento (SMF) datacenter region due to extreme weather. The unprecedented event resulted in the total shutdown of physical equipment in SMF,” Carrie Fernandez, the company’s vice president of engineering, said in an internal message to Twitter engineers on Friday.
Major tech companies usually have multiple data centers, in part to ensure their service can stay online if one center fails; this is known as redundancy.
As a result of the outage in Sacramento, Twitter is in a “non-redundant state,” according to Fernandez’s Friday memo. She explained that Twitter’s data centers in Atlanta and Portland are still operational but warned, “If we lose one of those remaining datacenters, we may not be able to serve traffic to all Twitter’s users.”
News of the outage comes a day before Peiter “Mudge” Zatko, Twitter’s former head of security who turned whistleblower, is due to testify before the Senate Judiciary Committee.
In his whistleblower disclosure to multiple US government agencies, first reported last month by CNN and The Washington Post, Zatko warned that Twitter had “insufficient data center redundancy” raising the risk of a brief service outage or even Twitter going offline for good.
During his time at Twitter, Zatko learned that “even a temporary but overlapping outage of a small number of datacenters would likely result in the service [Twitter] going offline for weeks, months, or permanently,” according to his whistleblower disclosure.
In a statement about the Sacramento outage, a Twitter spokesperson told CNN, “There have been no disruptions impacting the ability for people to access and use Twitter at this time. Our teams remain equipped with the tools and resources they need to ship updates and will continue working to provide a seamless Twitter experience.”
Data centers need “reliable water, power, humidity controls and refrigeration to live,” said retired brigadier general Greg Touhill, who served as the US government’s chief information security officer in 2016 and 2017.
“You want redundancy, not duplication, of your data locations to enhance your cyber resilience so you can take a punch from a natural disaster [or other event] that can take down a single piece of equipment or data center,” Touhill, who now heads the CERT Division at Carnegie Mellon University’s Software Engineering Institute, told CNN.
