Rejected automation?
Rejected automation?
Do the people making these things even know what they are doing? Maybe not.
Let’s share stories where your automation efforts have been rejected and you can’t quite understand why! Here’s mine.
I work in networking, a job that traditionally has been managed by a terminal and vendor-specific syntax. I used to hate the thought of automation when I was younger, why would something as important as networking be automated? I've made my career on being the clutch guy, troubleshooting complex problems, I love the art of understanding every cog in the machine and being able to visualize it. Then I started learning Python, and learning it was extremely difficult for me. It felt like an eternity between the time I poured myself into learning Python until the time I could actually make things people would want to use.
I was a supervisor working in a NOC. A NOC that had many beaurocratic requirements which got in the way of break/fix operational support, such as having to manually write an email to every customer that had an alarm, and calling every point of contact for that customer, as well as notifying the field techs of outages in their areas, and managing real operational issues. So many times I had to let real work slip through my hands because there were so many calls, so many cases, so many things to do.
Like most NOCs, we viewed alarms from SNMP. When something failed to ping, it would generate a loss of comms alarm. I had this idea to automatically notify the field tech for the specified area when a customer site was downed for more than 30 minutes, and that was a very complex thing to do, it required that I clean a lot of data... I spent days converting things like date strings into proper formatting. Once I presented it, I was told that we couldn't do this, because some political agreement made it to where the NOC was required to provide "positive contact" to other groups. I wrote it, tested it as proof of concept, specd out costs(MRC for the API I was using to send text messages was extremely cheap, it would cost the company about 6 dollars per month). Just like that, it was dead.
My director then wanted me to do something similar for our phone systems. Since our queue depended on user agent availability(your presence status), my boss wanted me to write a program to notify him if someone was unavailable for too long and the reason why. Yes, he wanted to know if someone took more than a few minutes to take a shit or get coffee.
That's when I learned, boomers only care about micro management, not efficiency.
Thanks for sharing. I did a bit of work for a NOC and know exactly what you mean about letting real work slip through your hands. I wasn’t directly responsible for managing the alarms, but it felt strange to be writing software streamlining the workflow. All the time I spent I felt like I could have just helped the technicians actually solving problems they faced in their day to day - to stop the alarms going off in the first place!
To be fair, most of the work that you have to do in a NOC is total bullshit. About 30% of the time you will be working on technical issues, and for most other people in the NOC, that would mean escalating the technical issues to me. Unfortunately, I had to earn the stripes, which means I had to work harder than everyone else, which meant doing their work as well as handling all escalations. Eventually, I was promoted to a supervisor for my efforts, but I did not want to be in a managerial role.
The real bulk of NOC work that is tiresome is the amount of alarms that are unnecessary. Managing SNMP is a nightmare, and configuring it properly involves a deep level of engineering knowledge. You can either tune the alarm board to only show certain alarms(which means parsing through many alarms to find out what is necessary and what isn't), or you make sure that devices that are onboarded are configured locally for what SNMP traps they will alert for. Typically, the devices' SNMP settings are not configured, so all alarms get sent to the SNMP server, and the SNMP server was never tuned to know which alarms it should show or it shouldn't, so there are alarms which don't really "mean anything" and alarms that "could potentially mean something if it's correlated with this other alarm," but most of the work is sifting through so much shit, to then have to troubleshoot a network issue for a network that was never documented in the first place.
Lol I love the term distributed monolith. It seems paradoxical but very much a thing I have experienced. I don't think I've ever been rejected when trying to automate something though, that seems crazy.
I wouldn’t quite classify this as automation, but I’ve been fighting for the past year for better scripting tools. I work on kiosk-style systems on customer networks. A big part of my job involves connecting to a device, pulling some logs, and running connectivity tests. I created a PowerShell script to automate this and submitted a KB so that others could use it, which sat in the approval queue for a few months before it got rejected.
I reached out to the team who rejected it and was told that all scripts need to be approved by a senior. I told them that a senior had reviewed it and approved it, and linked them the approval which they would have seen anyways. They then said that it also needed approval by the development team. “Okay,” I said. “What’s the process to get that approval? I don’t see any documentation about it.” After a number of emails to several different departments, I found that there is no process. I bugged everyone I could think of but got no replies; my manager got about the same.
In the 12+ months it took to come to that conclusion, I’ve made scripts to automate just about every common fix we apply. Right now most of our KBs instruct us to schedule downtime with the customer to fix things using the GUI, but that’s not necessary for 90% if these issues. I’ve submitted KB revisions for each of these, all of which have been rejected because they need an approval that doesn’t exist.
I’ve brought this up to my manager several times and gotten my seniors to back me up on how much time these scripts save. I’ve shown how effective these scripts are when we have system-wide critical issues where I save us hundreds of man hours of work. None of this has made any difference; apparently the development team just can’t be bothered to create a webform or whatever or even just answer emails.
Sounds like your manager needs to take it up the chain. Get one of the people that manages the money in on this. "The development team's refusal to engage in [standard process] is costing the company hundreds of man hours"
I’ve just given up at this point. I have my scripts and I’ll share them if I’m helping someone with an issue, but it was such a fight to even get them rejected that I don’t want to bother with that again on top of the rest of my work. If nobody in this chain that I’ve already gone through seems to care, and if developing these scripts doesn’t change my eligibility for a promotion (which I’ve been directly told it doesn’t), I don’t see the point in pursuing it any more.
Sounds like something you could fix with a repository. Teach your coworkers how to use git
I’ve thought about it many times but can’t find a good way to implement it. I don’t have access to the company’s GitHub or any shareable network locations. Don’t want to upload to my personal GitHub either since there is proprietary information in some of them. Right now I have them shared in a OneNote notebook that I manually update as I revise the scripts.
Estimating the savings with actual numbers and CCing some high level executive could have been worth trying, if you didn't already
It’s funny you would reply about that: I actually did escalate it again and I’m working on getting a process implemented. It’s like pulling teeth, but I’m determined to get this fixed. Luckily my manager is finally with me on this, so I’m making some real progress for once.
When the effort to maintain the automation outstrips the benefit. This can happen often in dynamic environments with many moving parts.
Yes I've personally argued against automation, too. Particularly when something is clearer to just be written down in a checklist-like format, to be followed by someone with domain expertise.