We didn’t think that should or would happen

“We didn’t think that should or would happen.” Duffy on the air traffic outage.

This seems like a fairly reasonable assertion to anyone who doesn’t actually implement changes to things that impact other people. What dude here fails to appreciate is that there aren’t many changes where we *expect* to take everything down. Most systems have redundancies. I can literally update and reboot one server in the system without anyone noticing. The difference between what we expect to happen and what DOES happen is why work is done during maintenance windows. Publicized so anyone who happens to be working at 2am knows who to call if things don’t look right.

I worked at a company that was planning an electrical upgrade at their data center. The electrician opened a floor panel to look at what he’d be doing. Literally just look. Somehow, dude dropped the quarter inch sheet metal cover into the opening, shorting some stuff out and dropping half of the servers that weren’t high enough priority for redundant power (also identified a few where redundancy power both went to the same place), had to take an ambulance to the hospital (he recovered and was fine). The data center ops guy – the one who declined all changes that weren’t in a maintenance window or accompanied by an emergency change and a critical issue ticket – kept that metal plate in his office as a reminder of why there were no exceptions. No matter how much someone might plead and assure him their change was so trivial.
Worked another place where they had a team removing the unused fiber. Contractors paid hourly, and the rate was higher for nights and weekends. Of course they wanted to save money and do this at 2 in the afternoon instead of 2 in the morning. And they found every poorly documented fiber connection into SAN shelves.
These changes were approved by immature change management structures who took “Nothing should happen, we’re just removing the unused fiber” as a 100% accurate depiction of reality. That took “I’m just opening this panel and looking over some wiring” as  immune to any accidents. In both companies, changes were made. The criterion for approving a change outside of the agreed upon maintenance window became much stricter. The “no impact expected” checkbox meant very little in deciding if the change would be approved or not.

For any organization that is approving non-critical changes during peak usage, hopefully this is a prompt to make some changes without having to take out dozens of your own systems. For the rest of us, though, it’s a good reminder why change management exists. Why it is a good thing someone is rejecting your change and requesting it be rescheduled during a maintenance window. To have some appreciation for the folks to review these changes for impact and handle end user notification. I think most of us who have worked in IT for a while know the feeling the guy who plugged in the Gear of Doom had. Having a couple extra people check your plan is a wonderful contribution to never feeling that again.

Leave a Reply

Your email address will not be published. Required fields are marked *