CenturyLink and 911 National Downtime [was: ATT Crash]

[grarpamp]

https://www.reddit.com/r/networking/search?q=centurylink&sort=relevance&t=week

https://www.reddit.com/r/networking/comments/a9z6tb/centurylink_outage_west_coast/
https://www.reddit.com/r/technology/comments/aa030y/centurylink_customers_reporting_major_internet/
https://www.reddit.com/r/centurylink/comments/a9zwkx/centurylink_down_all_over/

https://downdetector.com/status/centurylink
https://status.ctl.io/

https://i.redd.it/dsd0r48tnw621.png

"
Spoke to CenturyLink this morning about a downed 10G LAN PHY in WA,
the guy I talked to said they had a major DWDM outage causing problems
all over the west side of the country. Their own internal ticketing
system is currently down, ControlCenter is unreachable from various
locations on the west side, all kinds of problems.

They were working with their DWDM vendor, weren't able to give me an ETR.

6:30pm Eastern, I've repeatedly tried to reach 3rd level escalation
lines xxxxxxxxx or xxxxxxxx however both numbers were flooded and
indicated all services are unavailable upon each attempt and
ultimately was unable to get through. Called the 4th level escalation
lines xxxxxxxx or xxxxxxxx and spoke with Director of operations - who
indicated that they still do not have an ETA, however assured me that
they have engaged onsite hardware vendor support on all fronts to
solve the problem and restore service.

    Efforts to regain visibility to sites in Atlanta, GA and Chicago,
IL remain ongoing. Once visibility has been restored the filter will
be applied to limit communication traffic between sites which was
causing CPU spikes that in turn prevented the devices from functioning
properly.

​From the call I'm on, I've been told they had lost complete
management access due to these issues and had to wait for someone to
be on site to resolve it. That's pretty extreme.
"