Tag: outages

We didn’t think that should or would happen

“We didn’t think that should or would happen.” Duffy on the air traffic outage.

This seems like a fairly reasonable assertion to anyone who doesn’t actually implement changes to things that impact other people. What dude here fails to appreciate is that there aren’t many changes where we *expect* to take everything down. Most systems have redundancies. I can literally update and reboot one server in the system without anyone noticing. The difference between what we expect to happen and what DOES happen is why work is done during maintenance windows. Publicized so anyone who happens to be working at 2am knows who to call if things don’t look right.

I worked at a company that was planning an electrical upgrade at their data center. The electrician opened a floor panel to look at what he’d be doing. Literally just look. Somehow, dude dropped the quarter inch sheet metal cover into the opening, shorting some stuff out and dropping half of the servers that weren’t high enough priority for redundant power (also identified a few where redundancy power both went to the same place), had to take an ambulance to the hospital (he recovered and was fine). The data center ops guy – the one who declined all changes that weren’t in a maintenance window or accompanied by an emergency change and a critical issue ticket – kept that metal plate in his office as a reminder of why there were no exceptions. No matter how much someone might plead and assure him their change was so trivial.
Worked another place where they had a team removing the unused fiber. Contractors paid hourly, and the rate was higher for nights and weekends. Of course they wanted to save money and do this at 2 in the afternoon instead of 2 in the morning. And they found every poorly documented fiber connection into SAN shelves.
These changes were approved by immature change management structures who took “Nothing should happen, we’re just removing the unused fiber” as a 100% accurate depiction of reality. That took “I’m just opening this panel and looking over some wiring” as  immune to any accidents. In both companies, changes were made. The criterion for approving a change outside of the agreed upon maintenance window became much stricter. The “no impact expected” checkbox meant very little in deciding if the change would be approved or not.

For any organization that is approving non-critical changes during peak usage, hopefully this is a prompt to make some changes without having to take out dozens of your own systems. For the rest of us, though, it’s a good reminder why change management exists. Why it is a good thing someone is rejecting your change and requesting it be rescheduled during a maintenance window. To have some appreciation for the folks to review these changes for impact and handle end user notification. I think most of us who have worked in IT for a while know the feeling the guy who plugged in the Gear of Doom had. Having a couple extra people check your plan is a wonderful contribution to never feeling that again.

Centralization

The potential for noticeable, widespread outages is an unappreciated component of centralization. I started using Exchange in 2000 as a tier 3 sys admin at a Fortune 500 company. Big company everyone’s heard of, but you wouldn’t hear about it when our email system went down for a day. If you were actively looking at buying from us, you might get a call from your sales guy letting you know he’s going to need a little extra time on a quote. Or you might get the email from an “odd” address (the sales guy’s gmail account or whatever). But it wouldn’t be national news. It wouldn’t be anything that everyone knew about.

I worked a major outage every couple of years from 2000 through 2015 when we moved to the hosted Exchange service. I sat next to the network infrastructure team. They’d have site outages that impacted a single office with a couple hundred people working there. Inside the company, even that wasn’t general knowledge. Every couple of years, they’d have something massive melt down. Someone would have an upgrade go sideways and a quick three hour task turned into a full weekend with dozens of people working on restoring service. Even cyber attacks. Worked at a place that’s been hacked. It made the news, but like “third page, under the fold” kind of news.

Fundamentally, there isn’t anything different going on today. It’s just that we don’t have ten thousand companies with ten thousand email systems, ten thousand authentication directories, ten thousand file servers, ten thousand networks. We have ten thousand companies of which at least 8,000 of them have all their stuff in Google, Microsoft, or Amazon. Something goes down, you notice. That’s true of email systems, authentication systems (federated auth, if you “log in with Google!” to all of your favorite sites, you lose access to those sites when Google’s auth framework goes wonky), networks, etc.