Thread by @gravislizard

"shadow IT" is the worst condemnation of a companys IT department. like if it happens it means you fucked up bad enough people couldn't do their job

friend asked me to post an explanation of what shadow IT is so i'm gonna give you an example

about eight years ago my company had a problem: new VOIP phones would get shipped out configured to point to our servers, and would just arrive ready to use, but if a customer factory reset one, or otherwise messed with it, they could lose those settings.

there was no solution to this, because - oops! - turns out all the phones, everywhere, used the same password to talk to our system, and we couldn't tell customers that password. the engineering-approved solution was a miserable, 30 minute process

this process only worked if the customer had one of our special internet routers on site, which many did at the time. but if they didn't ,you were just fucked. literally, there was NO SOLUTION. this was the case for YEARS and nobody did anything.

agents would say "I can't actually fix this. it's impossible. what do I do?" and management just shrugged at them as far as i know. i think sometimes phones got shipped back to the office or agents just gave the customer the password, an absolute security violation.

but what do you DO? what's the solution in that situation? the customer is upset for a really good reason, and management is shrugging at you. the one option available to you, you're told you can't do.

so S, my predecessor, got fed up with this. he scrounged up an old PC, installed a fileserver on it, put a configuration file on it that contained nothing but a pointer to the correct server *with the password*, and wrote up instructions for the support teams

the support team would then instruct customers to put in the server address with no creds, the phone would connect, get the URL and creds, flip to the correct server and be off to the races. and the customer never saw the password.

of course, they could then go to their web browser, put in that URL and the mac address of their phone, and see the password. but they'd have to know to do that.

the machine was also firewalled and we had to open a pinhole for every IP address we did this with so it wasn't WIDE open, but anyone sniffing on that network would absolutely have caught this, and we have evidence I think that this happened at least once

obviously this is deeply nonsecure. i mean, it was an HTTP server! sending the password for our ENTIRE config server in the plain!! terrifying! and what about the responsibilities and maintenance?

some random employee shouldn't be turning up a SERVER on the companys behalf!! where was this server hosted? a random internet connection we had for hardware testing. who got called when it broke? nobody. S fixed it the next day when he got in.

the bureaucracy is ALLERGIC to this but *there were no alternatives.* the solution offered to the support teams was nothing. eat shit and enjoy it. get yelled at by a customer with a perfectly good phone who you're forcing a 48 hour RMA on.

i inherited this system, and it was something nobody wanted to talk about. everyone assumed this problem was solved some other way, so whenever someone found out this was how we were doing it they'd go "wait, you can't" and we'd go "we are. stop us."

literally, S and I both asked to have this system taken over by the Systems team dozens of times. nobody would do it. they also didn't tell us to shut it down more than mildly, but we made it clear it wasn't going anywhere without a replacement solution.

Finally, a year or two ago, it got replaced by an official, Systems-managed tool, and it sucks ass. It's terrible. I hate it, everyone hates it, and it's because the guy they gave it to didn't want to make it.

That part at least is a personnel issue we won't get into, but the point is, it basically took a DECADE to get Systems to do something A) trivial B) their fucking job and C) a critical part of the companys product support infrastructure

I know it was trivial, because *I ran it*. it was a fucking webserver with a shell script that called ufwadd and copied a file to a directory. It was two hours of work on the outside that nobody considered important.

There were MASSIVE security problems here, yes, but they were caused by an architectural issue that was essentially unfixable. Using one password for all your voip phones is a really bad idea! But you can't just pretend you didn't do it!

They caused this problem, then refused to solve it in any way that wasn't hellish for the support and provisioning teams. So my friend solved it by adding a network element that IT knew nothing about, and that's shadow IT.

The problem, of course, is that sometimes an employee "helps" by doing things that IT should have done, and other times they "help" by introducing a security hole you can drive a ship through.

And if you had a process for distinguishing the two, it wouldn't be shadow IT, it would just be someone doing a side job for the IT department.

i would love to say that the moral of the story is "listen to your users or they will go over your head" but the fact of the matter is that my experience was caused by personnel failures at multiple levels and there's not much you can do about it

I don't think competent and properly staffed depts necessarily end up with this problem at all, but idk. There are probably examples that you could say technically fit in the box.