“On September 5th, Twitter saw the loss of the Sacramento Data Center (SMF) area due to severe weather. The unprecedented event led to the complete shutdown of physical equipment at SMF,” Carrie Fernandez, the company’s vice president of engineering, said in a statement. An internal message to Twitter engineers on Friday.
Big tech companies usually have multiple data centers, in part to ensure their service stays online if one of them fails; This is known as redundancy.
As a result of the Sacramento outage, Twitter is in a “non-redundant state,” according to Fernandez’s note on Friday. She explained that Twitter data centers in Atlanta and Portland were still operating, but cautioned, “If we lose one of the remaining data centers, we may not be able to serve traffic to all Twitter users.”
The memo continues to block unnecessary updates to Twitter’s product until the company can fully restore Sacramento’s data center services. “All production changes, including deployments and releases to mobile platforms, are blocked, except for those required to address service continuity or other urgent operational needs,” Fernandez wrote.
The restrictions highlight the apparent fragility of some of Twitter’s platforms, an issue that Peter “Mudge” Zatko, Twitter’s former security chief turned whistleblower, raised in a statement sent to lawmakers and government agencies in July.
In his disclosure to the whistleblower, first reported by CNN and The Washington Post, Zatko warned that Twitter did not have an “inadequate redundancy in the data center” which increased the risk of a brief outage or even the possibility of Twitter shutting down forever.
Even a temporary but overlapping outage of a small number of data centers is likely to lead to service [Twitter] not online for weeks, months, or permanently,” according to Zatko’s whistleblowing disclosure. (Twitter has criticized and defended himself extensively against the allegations, saying the disclosure paints a “wrong narrative” about the company.)
News of the data center outage comes a day before he is to testify before the Senate Judiciary Committee.
In a statement about the outage in Sacramento, a Twitter spokesperson told CNN: “There have been no disruptions affecting people’s ability to access and use Twitter at this time. Our teams remain equipped with the tools and resources they need to ship updates and will continue to work to provide a Twitter experience. smooth”.
Retired Brigadier General Greg Tuhill, who served as the US government’s chief information security officer in 2016 and 2017, said data centers need “reliable water, electricity, humidity controls, and cooling to live.”
“You want to duplicate, not duplicate, your data locations to enhance your cyber resilience so you can weather a natural disaster [or other event] Tohill, who now heads the CERT division at Carnegie Mellon University’s Software Engineering Institute, told CNN:
“Beer fan. Travel specialist. Amateur alcohol scholar. Bacon trailblazer. Music fanatic.”