Category: Business

Simplified Outage Reporting

We’ve lost power the past few days — I think that’s because the forestry crew is clearing a few trees that are down on the power lines, but I don’t know so we report the outage. First Energy has a really cool way of reporting outages — you use SMS to register your phone with your account. Once the phone is registered to an account, you can text “OUT” to their short code. There’s a short code to register for outage notifications, status alerts. So when the power went out today, it took about three seconds to report the outage. No walking through the IVR to report the outage online. I hope to see more companies doing this in the future.

Agile Methodology Is Not Anarchy

For the past several years, my employer has been moving toward an Agile development methodology. There are some challenges when mapping this methodology into operations because it’s not the same thing; but, surprisingly, those are not where I have experienced challenges. The biggest challenge during this transition is some of my coworkers seem to think the methodology is that there are no rules.

A friend of mine, a fairly eccentric history professor, used to say that a little knowledge is a dangerous thing but you’ve got to emphasize LITTLE. And it seems like we’re encountering a situation where Phil’s emphasis holds true: the only thing garnered from from Agile training is that the documentation and process from waterfall projects are no more. But breaking away from the large-scale view of a P.R.O.J.E.C.T. for Agile development is a bit like breaking a monolithic application out into microservices — it still needs to do all of the same ‘stuff’, it just does it differently. And there are still policies and procedures — even a microservice team is going to have a coding standard, a process for handling merges, a way of scheduling time off, and some basic idea of what their application needs to accomplish. Sure, the app’s design will change incrementally over time. But it’s not an emergent property like chaos/complexity theory.

Maybe the “what Agile means to me” mentality comes from failing to clearly map a development methodology into an operations framework. Maybe it’s just a good excuse to avoid components of work that they do not enjoy. To avoid “agile operations” becoming “no boring planning stuff!!!”, I’ve outlined ways in which the Scrum methodologies the company wishes to adopt can be used to streamline operations. It helps that our group is reorganizing into an operational/support group and an architecture/design group — I see a lot of places within the operations team where Scrum approaches make sense.

Backlog — prioritizing the ticket queue like a backlog and having support staff constantly pulling from the top of the list — not only is this an awesome way to avoid the guy who scans the queue for the easy jobs, but it ensures the most important problems are being resolved first. A universal set of stakeholders does not exist for the ticket queue — someone whose ticket is ranked fifteenth on the list may disagree, and they are welcome to add details explaining why the issue is more impacting that it seems on its face. But 90% or more of our tickets are “Sev3” — which basically means both “we want it done ASAP” and “it isn’t a wide-spread high impact outage”. Realistically, dozens of tickets do not have the exact same time constraint and impact. There is extra work for management in converting a ticket bucket into an ordered backlog, but the payoff is that tickets are resolved in an order that correlates to the importance of the issue. In addition to the ticket queue, routine maintenance tasks will be included in the backlog. And prioritized accordingly.

Very short sprints — while developers moving from Waterfall to Agile might start from a month (or two) long sprint and trim weeks as they evolve into the process, operations starts from the other end of the spectrum. Our norm is to grab a ticket, sort it, then look at the queue and grab another one. We are planning for hours, maybe a day or two. This means we might establish application access on Tuesday that isn’t needed until next Monday. Establish a sprint that lasts a week, and use the backlog to get tickets that have lower priority (either because the impact is lower or because resolution is not needed for a week) included in the sprint. Service interruptions, SEV1 and SEV2 tickets, will occur and should be assumed in the sprint planning (i.e. either take enough work that you think it will just get done with no service interruption tickets and accept that some tickets from the sprint will be incomplete or leave some space for service interruption tickets and have staff pull “bonus” tickets from the top of the backlog if they have no work toward the end of the sprint).

Estimation — going through the tickets and classifying each incident as a quick little task, something that will take a few hours, or a significant undertaking facilitates in sprint planning. It’s difficult to know how many tickets I can reasonably expect to include in a sprint if I cannot differentiate between a three minute config change and a three day application rollout.

Multi-tasking — Implementation, support, and ticket resolution tasks are no longer a big bucket of work that individuals attempt to multi-task to complete. There are distinct tasks that are completed in series. Some tickets require information from the user; put the ticket on hold until a response is received and move on to the next unit of work.

Velocity — historic data based on time estimates cannot be generated, but simple number of tickets per week pre- and post- can certainly be compared. And going forward, ticket counts can be weighted by estimation values.

Stand-ups are a bit of a mental sticking point for me. I can conceive the value of spending a few minutes reviewing what you’ve done, what you plan on doing, and ensuring there is a ready forum to discuss any sticking points (maybe someone else has encountered a similar situation and can offer assistance). Stand-ups could include a quick discussion of any priority shifts (escalations, service interruptions) too. *But* my experience with stand-ups has been the attendance test variety — stand-ups that were used to hurt individuals who didn’t make it to the office by 08:00. Or those who weren’t around at 16:50. I don’t think it’s reasonable to ask someone who got into an issue and worked until 7P to show up at 8A the next day. I also don’t think it is reasonable to expect someone who came in at 6A to continue working until 5P. Were a stand-up scheduled in the middle of the day, I might feel differently about them.

DevOps Alternatives

While many people involved in the tech industry have a wide range of experience in technologies and are interested in expanding the breadth of that knowledge, they do not have the depth of knowledge that a dedicated Unix support person, a dedicated Oracle DBA, a dedicated SAN engineer person has. How much time can a development team reasonably dedicate to expanding the depth of their developer’s knowledge? Is a developer’s time well spent troubleshooting user issues? That’s something that makes the DevOps methodology a bit confusing to me. Most developers I know … while they may complain (loudly) about unresponsive operational support teams, about poor user support troubleshooting skills … they don’t want to spend half of their day diagnosing server issues and walking users through basic how-to’s.

The DevOps methodology reminds me a lot of GTE Wireline’s desktop and server support structure. Individual verticals had their own desktop support workforce. Groups with their own desktop support engineer didn’t share a desktop support person with 1,500 other employees in the region. Their tickets didn’t sit in a queue whilst the desktop tech sorted issues for three other groups. Their desktop support tech fixed problems for their group of 100 people. This meant problems were generally resolved quickly, and some money was saved in reduced downtime. *But* it wasn’t like downtime avoidance funded the tech’s salary. The business, and the department, decided to spend money for rapid problem resolution. Some groups didn’t want to spend money on dedicated desktop support, and they relied on corporate IT. Hell, the techs employed by individual business units relied on corporate IT for escalation support. I’ve seen server support managed the same way — the call center employed techs to manage the IVR and telephony system. The IVR is malfunctioning, you don’t put a ticket in a queue with the Unix support group and wait. You get the call center technologies support person to look at it NOW. The added advantage of working closely with a specific group is that you got to know how they worked, and could recommend and customize technologies based on their specific needs. An IM platform that allowed supervisors and resource management teams to initiate messages and call center reps to respond to messages. System usage reporting to ensure individuals were online and working during their prescribed times.

Thing is, the “proof” I see offered is how quickly new code can be deployed using DevOps methodologies. Comparing time-to-market for automated builds, testing, and deployment to manual processes in no way substantiates that DevOps is a highly efficient way to go about development. It shows me that automated processes that don’t involve waiting for someone to get around to doing it are quick, efficient, and generally reduce errors. Could similar efficiency be gained by having operation teams adopt automated processes?

Thing is, there was a down-side to having the major accounts technical support team in PA employ a desktop support technician. The major accounts technical support did not have broken computers forty hours a week. But they wanted someone available from … well, I think it was like 6AM to 10PM and they employed a handful of part time techs, but point remains they paid someone to sit around and wait for a computer to break. Their techs tended to be younger people going to school for IT. One sales pitch of the position, beyond on-the-job experience was that you could use your free time to study. Company saw it as an investment – we get a loyal employee with a B.S. in IT who moved into other orgs, college kid gets some resume-building experience, a chance to network with other support teams, and a LOT of study time that the local fast food joint didn’t offer. The access design engineering department hired a desktop tech who knew the Unix-based proprietary graphic workstations they used within the group. She also maintained their access design engineering servers. She was busier than the major accounts support techs, but even with server and desktop support she had technical development time.

Within the IT org, we had desktop support people who were nearly maxed out. By design — otherwise we were paying someone to sit around and do nothing. Pretty much the same methodology that went into staffing the call center — we might only expect two calls overnight, but we’d still employ two people to staff the phones between 10P and 6A *just* so we could say we had a 24×7 tech support line. During the day? We certainly wouldn’t hire two hundred people to handle one hundred’s worth of calls. Wouldn’t operations teams be quicker to turn around requests if they were massively overstaffed?

As a pure reporting change, where you’ve got developers and operations people who just report through the same structure to ensure priorities and goals align … reasonable. Not cost effective, but it’s a valid business decision. In a way, though, DevOps as a vogue ideology is the impetus behind financial decisions (just hire more people) and methodology changes (automate it!) that would likely have similar efficacy if implemented in silo’d verticals.


Sitting is the New Smoking

Some company official posted an internal article titled “Sitting is the New Smoking” to tell us all how bad sitting for prolonged periods of time can be for your health. While they make suggestions for using your break to do some exercises or suggest cube-exercises … frankly, they’ve designed a job that requires sitting for prolonged periods of time.

Some people have standing desks. Not all. Not most. My understanding is these things were purchased as accommodative equipment the company had to purchase based on medical need. If sitting is as bad for your health as smoking, did the company not just publish its own statement of medical need to support widespread purchase of standing desks?

Beyond near-term costs, though, the assertion brought to mind the Black Lung Benefits Act from nearly two decades ago. While mine operators may have been able to reduce exposure to coal dust, some level of exposure to coal is requisite in mining the stuff. A generally unavoidable environment based on the work being done caused a major medical problem that led to disability and death, and companies ended up shelling out disability payments and survivor benefits. It wasn’t quite the least they could possibly do to quell public outcry, but there are a lot of *’s on qualifying that let reasonable requests be denied or pushed off for years without retroactive payments. Even so, the payout is like eight grand a year per afflicted miner. And there are like 30k recipients (and something like 5k dependents, which can drastically increase the annual payout). That’s minimum two hundred forty million bucks in 2017. And it’s a LOT less now than a decade ago. There are nuances to determining the payer, but it is generally the mine operator most recently employing an affected individual. A significant portion of this money has been payed by mine operators.

Sitting at work is different from exposure to coal whilst mining coal. There’s no reason most jobs require sitting for hours on end. Historically there’s a component of elitism — a hundred plus years ago, low paying jobs were physically intensive, and it was a bit of an elite thing to be able to sit at work. Now the sign of affluence is a few spare hours a week to exercise, and sitting is just a norm no one has sought to change. If a company is aware of how bad sitting is for its employees, seems like said company would have a better defense against liability if they actively attempt to re-design their workplaces and jobs to avoid sitting. Sending out a mass mail telling you how bad something is or having a webinar to tell you how bad it is … but generally employing people to sit for hours at a time isn’t much in the way of due diligence. Routinely deploying standing desks, even in training classrooms, would reduce mandatory sitting among call centre staff. Walking meetings for one-on-one or small group sessions.


What, me worry?

Steven Mnuchin, one of Trump’s best people, is not worried about mass worker displacement due to automation. Said so at an event hosted by Axios. I’d love some of whatever he’s been toking.

In the near term (and evidently that’s all business execs or government types care about these days), sure automation and AI will drastically increase profitability. But I foresee the trend following a similar path as off shoring … great for individual businesses, but at some point capitalism mandates people have some money to buy the stuff and neither offshoring or wide-scale automation is sustainable. Offshoring at least provided alternate jobs for enough people to float enough debt to sustain the market near-term. We’ve got “knowledge workers”. But what percentage of those can be turned into AI programs? A significant number. I automate 80% of IT work. Chat bots could provide at least half of legal and medical consultations — the routine stuff. Robots make products, load the truck/train/drone that drives itself. Right to your door, or even inside if you have the Amazon lock. There aren’t a lot of jobs where some portion couldn’t be automated today. And budget cuts and productivity demands essentially require it. Some lucky few own doomed companies and profit for some time, another really lucky few are AI programmers and electronics engineers (although self-building AI/robots are totally a thing too). Maybe automation will beget a whole new industry that will provide good jobs for billions of people. Maybe the capitalist system will collapse and everyone will have more than they need (the Star Trek series, I guess). But I don’t know that I wouldn’t worry about the impact automation has on employment and the economy.

Drone Army

Over the weekend, when it was negative five degrees, our neighbor’s power went out in the middle of the night. Some trees along the line grew into the power lines and had been abrading the line for some time, and a handful of arborists had to come out and try to trim the tree back. In the dark. At negative five degrees. Not the most fun job I could imagine, and the ironic this is it was the same team that had been out in the summer to clear trees along a stretch of the power lines a bit farther down.

The problem, it seems, is that it’s terribly time consuming to have arborists walking along the line to see where things actually need to be cut. Instead they just hit every section once per unit time. Sometimes that’s a quick couple branches snipped in a hardwood grove. Sometimes that’s serious maintenance in softwood groves. And sometimes delta-time is too long for, say, our line of pine trees. And sometimes the team doesn’t do a particularly good job of trimming the trees.

Made me wonder about having drones fly along the line – you’d still need someone to drive out, and I’d recharge the batteries in the van/truck so they’d be ready to go when I got to the next site. A single person flying a drone over a stretch of power lines could generate more realistic work orders for the arborists – skip the bits that didn’t grow much, realize these pine trees are endangering the lines before you had to call out a crew on Sunday night. They could also run through the same line post-maintenance and verify the work was done well.

Customer Service And IT Automation

A 3D printer filament manufacturer, MakerGeeks, has been running a series of awesome deals since Black Friday. We placed an order for several of their their “grab bag” packages – which I assume to be production overruns and whatever isn’t selling. We want to make a few large prototypes – if it’s an amalgamation of oddball colours … whatever, it’ll still be functional. We can pay extra to select the colour once we’ve got a finished model file.

A few hours after placing my order, I got a mass e-mail saying essentially “we sold a lot more stuff than we expected, it’s gonna take a while to ship”. Wasn’t buying Christmas presents, so waiting a while … whatever. Two weeks later, I haven’t heard a thing from them. Odd. I sent a quick e-mail asking for someone to verify that my order didn’t get lost or something. And never heard back from them. Waited another week and sent a follow-up.

Checked them out on the BBB site and found out they’ve got a really bad reputation for non-existent customer service And not shipping ‘stuff’. Sent an e-mail to all of the contacts listed on the BBB site (the phone number is unanswered and rolls to a generic message). Another week with no response, and I filed a BBB complaint mostly to increase the number of people saying “these people don’t bother answering e-mail and suck at order fulfillment”.

Additional irony – I’d subscribed to their newsletter when we placed our order. The five weeks of no communication from the company did include an almost daily e-mail with information on their holiday promotion. So they’re not bothering to ship my stuff, but they’re actively soliciting new orders!?!

What bothers me, though, is that a simple automated job would be the difference between initiating a charge-back and waiting for my order to ship. There’s an order database somewhere. Pull a list of all open orders & send a message that says increasingly comforting versions of “we haven’t forgotten about you, we just haven’t gotten to you yet”. If it were me, I’d probably include something like “We currently have outstanding orders for 25,839 KG of filament that we’re working through. The machines are running as fast as they can, and we’re shipping 2,848 KG a day. We want to thank you for your patience as we work through this amazing volume of holiday orders.”. Actual message content is almost irrelevant. The point is a few dozen development hours would be saving orders and improving the company’s reputation.

Instead I get nothing. With no faith that the company will ship me anything ever … and since I don’t want to try disputing a charge six months after it was made (had problems with that before – prepaid a CSA membership through PayPal, waited eight months for the new cycle to start, but I wasn’t on their list and they claimed to have no record of my payment. Tried to dispute it through PayPal and was told the window to dispute the charge was up … but I didn’t know I wasn’t going to be part of the new year until the first delivery!), I presented my communication and their complete lack of response to the credit card company. About 24 hours later, the charge-back was completed.

Insurance And Medical Billing


There’s been a lot of talk about health care reform – years ago when the ACA was written, over the summer when the Republicans were working on a replacement bill, and again now that health care is trying to get slid in with tax changes. At no point has any politician addressed the real problems in health care costs: medical billing.

Scott went to his doctor for a routine checkup — a preventative service that is 100% covered by insurance. The doctor asked him if there’s anything else. He mentioned back pain, and his primary care physician referred him to a back pain specialist. We get the bill and he got billed for both the routine checkup and a medical appointment. Evidently, when the doctor asks if there’s anything else during a routine checkup … the answer is NO WAY IN HELL, otherwise you get billed a couple hundred bucks. Didn’t look at his back, didn’t prescribe any medication. Just said “yeah, you’ll need to see a specialist”.

The worst part is, in talking to the medical billing people, a doctor doesn’t know at what point a conversation will be deemed sufficiently in depth as to incur an additional code on the bill. There is absolutely no other situation where people would accept blindly accepting a service without knowing the charges involved. Could you imagine taking a University course and getting a bill at the end based on some financial department worker’s interpretation of how much interaction you had with faculty and educational resources for the duration of the class? A restaurant meal where the bill comes six months later and is based on the time you spent at the table, each interaction with a server, you chatted for a few minutes with the guy who brought the beer and that’s an extra fee because you discussed the IBU of their different offerings. Could have just said hoppy bitter flavor, but you used a technical term and incurred a consulting fee. Hell, you take your car in for service and they’ll provide an estimate before performing maintenance.

What I don’t understand is why we accept this billing model for medical services. I saw someone on SharkTank a week or two ago selling at-home medical testing kits. Her sales pitch wasn’t just the convenience (or privacy) of testing for medical problems at home. It was that there was a known cost for each test. You want to know your cholesterol levels? That’s 80$. Thyroid problem? Measuring TSH, TPO, free T3, and free T4 levels costs 150$. You pay in advance, you know how much it costs, and you don’t get a bill thirteen months later for services you never consented to receive.

What recourse do you have when the Cleveland Clinic screws up your appointment and you end up with expensive bills you didn’t anticipate receiving? Or a quick comment to a doctor garners another 150$ charge? Not a lot. Leave messages for their ombudsman who never returns calls. Pay the bill and appeal to the credit card company? Take them to small claims court?

Corporate Expense Reduction: Customer Service

We analyse calls to both our internal and customer-facing support desks all the time in an attempt to identify what causes the most calls. Theory being that fixing whatever-it-is would significantly reduce call volume and thus expenses. But do we ever analyze who is calling us? It may be a single problem that reoccurs. It may be ten little problems indicative of a bigger issue. It may be someone who is trying to avoid work and booking offline time on the line with the help desk (it’s been 20 years, but I used to generate a report for each call center manager of their director reports’ tickets & resolutions each week because it was so common a technique for a freebie 20 minutes off the phones and the manager didn’t have visibility into the ticketing system to determine if the calls were legitimate or not).

Multi-million dollar accounts (or senior company execs) get dedicated liaisons who walk issues through support channels. What if we had a small team of transient white glove support liaisons? People who have cultivated the same process knowledge and contacts that major accounts managers and exec support managers have. The top five callers for the week/month get rung up and their issue fixed. That may be calling around to get the proper support organisation looking at the issue. It may be replacing the thing that keeps breaking (replace the employee’s PC, replace the customer’s DLS modem). Even if you break even, there’s a much better experience with “yeah, I called in five times. But then so-and-so rang me up, took ownership of the issue, and fixed it” than “I have called eight times a week for the past month, had to explain the problem two dozen times, and these dolts finally managed to fix it”.

The other thing the metric could give you – and maybe a company would never use this – is how much a customer is *costing* the business. 800 charges, support staff man-hours … someone who has a 5$ a month account but uses ten hours of customer service time is costing the company money.

Corporate Expense Reduction: Energy

One of the things we’ve done with our home automation is tracking energy usage – partially because we want to size out a solar installation and the net metering in Ohio is awesome unless you produce more electricity in a rolling 12 month period than you use. So the installation has to be really close to your actual usage. But also because electricity costs money. Similar approaches may be beneficial to corporations. I’m using our 11 cent per kWh rate as an example. Actual rates depend on location and usage.

Does a company want to devote resources to “office automation” like we have home automation? Coupling motion detectors with smart outlets {or even just office schedules – if the last person is off shift at 7PM, dropping some device power at 8 should have no impact} to turn off power might save a lot in standby draw.

Even without home automation, companies can gather usage data to allow resources to be devoted to their biggest energy draws. The first step is identifying the big draws. We use Aeon Labs zwave clamp on home energy meters, but there are stand-alone energy meters. I’ve seen DIY Arduino based ones (, or high end Fluke devices with clamps do the same thing (@5k+ for the Fluke … that’s a bit of an investment, but if someone’s got an energy metering capable fluke for other work ‘stuff’ … they may just need the 10$ clamps). Whatever equipment – clamp it on one circuit in a panel for a few days. Get a number, move it to the next circuit. Eventually you’ve got daily usage numbers for different circuits and just need to look at what is on those circuits to narrow down potential saving points.

We found obvious stuff – HVAC uses a lot of power. If a company leases a building with outdated equipment, use firm numbers in lease negotiation. The HVAC draws x kWh per year which costs us y $. A middle-road new system should draw z kWh which means we’re spending some concrete dollar figure per year because this system is so old. The same information can be used to cost-justify upgrades/replacements for company-owned buildings. Measure usage on lighting circuits. An office with old ballasts and florescent bulbs – what they are costing to run tells you if switching to LED {and there *are* LED T4/8 tubes that don’t require fixture replacements} makes any sense.

But we also found things I would never have even considered if I made a list of all of our non-trivial electrical draws. 20% of our annual electrical usage is the septic aerator (it literally uses more energy than the geothermal HVAC system in a year). We can get it down to 11% of our projected usage by cycling the thing on during even hours and off during odd (or on/off in twelve hour chunks, or 4 on / 4 off / 4 on / 4 off / 4 on / 4 off … new aerators have scheduling and do this themselves). Now that septic aerator savings is only like 250$ a year. Not a huge amount of money, but it’s 250$ I would never have realized we were spending otherwise.

From an IT perspective – if a server supports wake-on-lan … does a backup server and tape library need to be running 24×7? If someone kicks off a restore, can it be powered up (adds a minute, but saves power whenever restores aren’t running) and can it be programmatically powered on maybe half an hour before its backup jobs are scheduled to kick. Then power back down when no jobs report as running or scheduled for x hours. As a company, we mandate that all computers be left powered on so patches can be deployed overnight. What if the nightly patch check-in then powered the computer down (either because there are no patches or after installation in lieu of a reboot)?

Or a printer — there is no need for the printer to be in standby mode for the 15 hours a day no one is around to print. Or the weekends when no one is around. Or company holidays. One of the fairly large Xerox printers we have draws a continuous 11 Watts in sleep mode uses 71 kWh each year between 17:30 and 07:30 M-F and all day Saturday and Sunday. Maybe 72 kWh if you add company holidays. That’s not quite eight dollars a year in savings (and power consumption won’t be 0 if the device can be woken remotely) – but saving 6$ per printer in a company with 2000 printers is 12,000$ each year. Some of the older printers don’t even have a lower power sleep mode and draw 95 Watts in standby mode – 620 Watts per year when no one is around, and just under 70$ in electricity. Even better – HP offers an auto-off / auto-on on activity feature that allows energy to be saved during working hours.

Are there intangible benefits to energy saving initiatives? Get into the automation side of energy savings, would some tech magazine profiling the effort (free publicity, and tech magazines are a good place to advertise a company offering network services)? Can companies form partnerships with geothermal / solar / wind / whatever manufacturers to get cheap installations + publicity? Sadly, in some markets that may not play well (what, you don’t want to burn coal!?!) … but it might not be seen as a negative if it approached as a “save money, do right by stockholders AND customers” message instead of a “green, save the planet, global warming is bad” message.