[CONTEST CLOSED] Share your IT horror stories! Win a treat!

[AmyReyes]

UPDATE: This contest has ended. Thank you all for frightening us with your spooky stories! Our randomly selected winners are: @Polymathink, @EJN, and @Madhan_kumar_G. Congratulations!

Don’t you just love fall? From the spicy smell of pumpkin pie baking in the oven, to the crunch of leaves underfoot, to the crisp chill in the air… it’s a great time to grab a blanket and cozy up by a fire. And now that October is here, Halloween is just around the corner. While we don’t have any tricks in store for you, there just may be some treats… read on for details of how you can secure your chance to win a pair of Cisco Meraki fuzzy socks.

‘Tis the season to be spooked! Before you dust off your spell books and break out your broomsticks, though, we thought it would be fun to kick things off with a little contest. I don’t know about you, but I love a good scare. 

So gather round, pull up a virtual chair by the light of our pretend campfire, and let’s tell some ghost stories! We want to hear your IT horror stories… Regale us with your tales of nightmarish networks, spooky servers, and petrifying permissions. What’s the scariest scenario you’ve encountered? 

How to win

To enter the contest, tell us your IT horror story in the comments below before 11:59 p.m. ET on Monday, October 31st. Be careful not to use any company names, and be kind in your tales. We will then randomly draw 3 winners to receive a prize.

Read on if you dare...

If you need a little inspiration to get you started, we sourced some stories from our own Cisco Meraki team. Enjoy!

The Tale of the Vanishing Emails

Adam Slater, Cisco Meraki Content Engineering Manager, shared a story from his early days of working at a VAR. “I was on a project to migrate a school’s network and servers to a new site. Back then, everyone had their own Microsoft Exchange servers for email. We needed to take the site offline for a little while to move it over, and there was this email service that would spool up and hold mail on a cloud app, then you could reconnect it to deliver mail to the server after. I moved everything, installed the new firewall and equipment, and … I misconfigured the firewall. I pointed the address of the email server to a database server that just happened to be not configured correctly. The same port was open that received email, so as soon as it came online, the service said ‘I can connect to the email server.’ It unspooled a week’s worth of email to the database server that said, ‘I don’t know what to do with this,’ and rejected all of the mail. There was no backup, so I lost an entire week’s worth of this school’s emails.”

After an hour or two of troubleshooting, Adam talked to the IT department and contacted the head of the school to explain what happened. Unfortunately, there was no way to recover the lost emails. Luckily, because he owned up to his mistake fast, he got to keep his job! It was a valuable lesson in ensuring that there’s always a backup.  

The Hard Drive on Haunted Hill

Lance Fiveland, Product Architect for Cisco Meraki Global Enablement, offered a harrowing tale of his own. “Early in my IT career, I was a post sales engineer and was tasked with setting up core servers to our top clients. On this one particular instance, I had the owner watching me like a hawk. I might say he was stalking me, in keeping with the horror theme. After several days of setting up his servers and getting all of his clients up and running, we did a half day of training and made sure everyone was happy and things were humming along nicely.  I departed and went home that weekend with a strong sense of accomplishment. No sooner was I resting on my laurels than a 911 call came across our desk. It is the owner yelling and screaming that the whole network is down, and he is demanding something be done immediately. After several hours of trying to walk him through diagnostics to no avail, I jumped in my car and made the long, arduous journey to the client.

Upon entering the premises, I was met with the zombie horde of disgruntled employees and customers. I made a quick haste to the server room to see what was wrong and get them back up and running as soon as possible. Well to my horror, I noticed the owner had taken several brackets and very long screws and bolted the servers to the desk. What he didn’t realize at the time is he drove the screws into all the hard drive platters and destroyed everything in the process.  He wanted to stop anyone from stealing the equipment and, in the process, almost got himself killed but ended up just killing the servers instead.  When I tried to explain to him that this isn’t something covered under any warranty, he was not very pleased but had little recourse.

I did end up spending that weekend installing several new servers to satisfy the client and made sure the owner did not get anywhere near us or the server room again.  So all in all a happy ending and a pretty large sale.” 

oOoOo terrifying! Luckily, all’s well that ends well in that tale. We hope you enjoyed these scary stories from our team, and can’t wait to read what you all have encountered!

 

TERMS & CONDITIONS

• Limit one entry per community member
• Submission period: Friday, October 21st, 2022 at 12:00pm EDT through Monday, October 31st, 2022 at 11:59pm EDT
• Prize will be a selection of Meraki swag with value not exceeding USD 50.00
• Official terms, conditions, and eligibility information
  

[asgerbjarke]

A client experiencing connectivity issue. The resolution was to correct the default gateway IP to an IP address in the same subnet.

[DarrenC]

On a dark and stormy night a few years back...

I got a request to troubleshoot a PC that had died.

The user was shall we say less that Martha Stewart for keeping his workspace tidy so could be many different things.

The need was urgent because not only was the user down, the group he worked in was down too.

Coming over it was a dark and scary cubicle. Rain was coming down outside and a late fall day not long before Halloween. The power outage, weather and decorations set the frightful stage to make it a dark and rainy night... well... late afternoon.

Facilities soon had the breakers back on again and I was looking over the employee's workstation stuffed into the back corner.

It had overheated a few times and the last tech had taken the side panel off and left it leaning against the case for better air flow.

Moving some of the clutter out of the way I unwrapped the mystery... moving the panel to the side to get a better look inside... it was a rat's warren inside, dust, droppings, dust bunnies... and... a literal rat. 

unbeknownst to the user his dietary habits and leftovers had attracted the other IT and cord loving bunch who had setup shop and conveniently removed one rat infestation challenge by connecting the terminals, shorting out the station, and tripping power leaving the group in the dark and a slightly smoking corpse.

Needless to say, the employee soon had a brand-new workstation, the mystery was solved. If it wasn't for those pesky kids and their snacky habits.. they would have gotten away with it too.

-Darren Cannon

[alemabrahao]

A long time ago a customer was trying to print a document and we found that there was no printer installed we informed him about It and he asked, do I need a printer to print? 😅

[Polymathink]

I had just been hired as a consulting IT Director. My first task was to review everything in place and to comment on what needed to be done. Concurrently, I was told that we would under no circumstances be changing anything.

It turned out the servers were generations behind, operating systems aged, and desktop systems antiquated. There may as well have been cobwebs holding the entire lot together. Never mind security, the front door was practically unlocked.

Upon bringing this to the attention of the Firm Administrator, I was told, "Well, no one is interested in us; we're not IBM", yet they must have thought we were Oracle because the internal IP address scheme was not private but instead set to a public address range assigned to Oracle.

This place was haunted by the spirits of poor IT decisions past and present. We were destined to a doomed future if attitudes could not be changed, and fast! So I set about making the small changes I could. First was insuring we had backups, closing outward facing ports, shoring up Citrix security, running updates, adjusting firewall rules.

With the low hanging (and no to low cost) fruit picked and having earned a bit of trust and now hired full time, it was time to move on to software licensing, aging hardware, and outdated firewalls, bringing Meraki and MDM into the picture and to bear on the previously untamed hybrid BYOD and firm issued endpoints.

The ghosts of IT past finally fell once we renewed the iron (chains) in the rack and began our move to the cloud. Meraki PoE switches were installed to support our eventual move to IP telephony from a decrepit Avaya system. Meraki meshed WiFi access points unshackled (towns)folk from their desktops, their pitchforks giving way to tablets. We were able to segment the network to keep visitor device traffic apart from internal devices. No longer were we at the mercy of the banshee-like updates of visitor devices bringing down external connectivity.

We've made it well past midnight in this haunting and can just see the dawn through the clouds. We've got a spell to go until we reach full daybreak, but we're well on a way to a rosy future.

[TMRoberts]

Oh I had an interesting and spooky time at the one place .... worked with security guards who complained they thought someone had broken in while they were on duty.

So, we ended up going through camera footage of the nights they said they thought it happened. 

You see the guard walk down some stairwell (internal so no wind), and as they walk past the one door it pops open and closed .... no one on either side. Dude freaks and rattles handle ... maglocked.

Then he opens door peered through, still no one there.

Walks away shaking head .. gets near the next floor... that door pops wide open (not like a little magdoor pop, WIDE open) ... 

Sec guard runs down last steps and looks through, still no one there.

We went through all the cameras at the doors and nearby hallways ... not a soul was in the building .... except this guard.

Next day, different guard mentioned earlier in week they had been in a stall in restroom ... door opened .... again no one else in building when we checked the cameras.

Glad we moved buildings two months later!!

[RomanMD]

Once upon a time, there were network devices which had CLI interface ... 😬 The peoples administrating them were called Network admins... then a Merakle happened and all those admins left without a job 🤣

[OffaMiWifi]

Once upon a midnight dreary, I upgraded a Windows 98se machine to Windows Millennium. AAAAAAH!!!!

The end.

[Hezinuk]

Customer: We think we have an issue with our switch! 

Me: Let me have a look 

 

[Inderdeep]

Story 1: A task was given to one of my colleagues in my earlier org where he needs to configure the access list for a specific interface to allow communication on the core switch in DC (Nexus 7K). He didn't take the prechecks and unfortunately delete some other Access lists with other configurations turns to be a messy situation. 

Story 2: Another incident where I saw one person adding the VLAN to the trunk interface and he ran the wrong command which replace all the VLANs from the trunk interface and communication in the DC stops for that specific L2 links. 

[ekramer]

I worked in a town with a community pool. The switch closet was not airlocked from the chlorinated air; the switch expired faster than an out-of-shape lifeguard.

[Will07]

Once upon a time (just yesterday).....and yes, this is a true story. 

We received an email from Meraki and the subject line goes "Your Meraki order has shipped - serial#....We're pleased to provide you with this information for your Cisco Meraki bill...".

I thought it would be one of those phishing/scam emails because I wasn't aware of any new purchases from Meraki. I was cussing about how our anti-spam didn't work and let this one slip through. Anyhow, I decided to read this so-called spam and it turned out it was actually from Meraki! The email shows the correct company name, PO order#, the dashboard license key, but with an unknown ship-to address and an unknown recipient contact information. I still need to check with the company to make sure we didn't get billed for something we never purchased for. That's actually spooky!

[EJN]

The plans were set. Camping after school on a Wednesday for 4 nights of camping in the Florida Keys. Car loaded, classes done for the day, and no open IT issues to resolve. Let's bring the laptop just in case to check email at night... except one problem, time is short and there isn't any room in the car for the laptop backpack. No worries... just slide it on top of all the camping gear on the rear of the car.

Leaving school, let's stop for gas. Open rear door, laptop almost falls, but no worries, set it on top of roof while filling with gas. Mind is all over the place trying to rush. Gas done, back in car... laptop still on roof. As the person who will remain anonymous for now pulls out of the gas station and into the onramp for the highway, all of a sudden a sliding sound and a thump. What was that?! The F*&^(*@G laptop!!!

First thought, oh well, nothing that can be done. It's broken and unsafe. 30 seconds later... why not make a uturn at the next exit and see if we can at least recover the data on the SSD. Sure enough... there it is right in the middle of the lane in the onramp. Pull to the side, wait for traffic to pass, and grab it. Success! For sure the screen is broken... nope. Certainly it won't turn on... it did. It won't boot... it did. Just a bit of separation on the screen and a handful of scratches on the lid.

Lesson learned... don't rush. 

Best part of the story... the anonymous person is me, the Director of IT at the school... the same person preaching handle with care to all the teachers.

[van604]

was asked to look at a wireless access point issue, and determined that it was a bad config file so i asked the onsite tech to do a pin hole reset on the WAP, i described what it looks like and i got a lot of yeah yeah, i got it.

next thing i know, he reset the the switch and blew away the config file, spend the next 2 hours rebuilding the config. nothing like pressure doing it remotely too , thank goodness for cloud managed devices now.

[Brash]

So excited to read these!

The one which I'll always remember was back when I was working in support. A call came in from a customer with a fairly sizeable network freaking out about a full network outage.

Jumping into the network we found all traffic being black holed. Following the routing it seemed like everything was being routed to their VMware cluster. Asking the usual questions, the customer confirmed they weren't using VM's for any type of routing.

Logging into the VMware cluster, we identified the VM which all traffic was being directed to. Shutting down the VM resolved the issue and the entire network was restored.

After discussing further with the customer, we found that the cause was that they had spun up GNS3 on a VM with a replica of their network. It ended up route poisoning the entire network.

The customer was relieved and it became a good story for the pub.

Tldr: Customer called in for support with a full network outage. The culprit was a GNS3 VM route poisoning the production network.

[BlakeRichardson]

Years ago my old boss had a very important customer come and see him about buying a new laptop. The customer ran a large consultancy company and everything ran off his Palm Pilot. 

I have the customer existing Mac laptop and his new Mac laptop connected booted off an external disk. My plan was to erase the disk on the new laptop (this was before MacOS has a migration assistant) and clone the system and data. I'd done this numerous times and had never had a problem. 

This time however though my boss who had no technical skills decided he would start the process and did so by erasing the drive on the old laptop destroying all of the customers data. Unaware he had done this he started the process and went back to his office. 5 minutes later I have to explain to him his mistake and I watched the blood drain from his face and his mouth hit the floor.

An hour later the customer arrived and some words were had, thankfully my boss owned his mistake and didn't try to throw me under the bus.

[almightythor]

Happy Halloween!

[Naveed]

i got a request through email that he wants some cisco items. i found that the company is ligit and the email address is official as well. i send him the quotes and then try to call him to talk to him. i called the number. the number given in email signature was same as mentioned on their website. when i check only last digit is different other than that the number is same. that number was not working. so i dialed the number and try to connect the person who has the email address. after struggle of 4 days. i got connected to that person. and he said that it is my email, but i didnt send you email. i asked him that the quotes he was asking me for 50k plus. he said please dont process any order. it is definitely my email but i am not sending any email. 

so i saved zones a scam. 

[deadpool]

The newest is from the new team taking over support. I was working on a migration when a Teams message pops up:

- Are you available for a quick call to troubleshoot a VPN?

- Sure, go ahead. - I said.

I joined the call. The shared screen showed a Cisco router prompt just after logon. Engineer from the new support team took the lead, and asked the question which surprised and terrified me at the same time:

- How do I see the configuration on this Cisco router?

[SeanBegley86]

Hi Amy,

This is my IT horror story and one that I have told many times.

I was working for a charity as an IT helpdesk Technician, learning the ropes. I was in the role for just under a year. I was informed that in a few weeks the Charity was hosting a major meeting of top doctors and scientists in their field, Epilepsy and neurological conditions in young children. The meeting was to be held in a conference room in one of the country's state-of-the-art medical centers on-site and mediated by a professor flown in from Australia especially. 

The room was filled with top professionals from all over the world when I got a call:

'Hi Sean, One of the projectors in the main conference room is alerting its filter needs cleaning. Could you pop down at 11 and give it a clean as we'll be on a tea break then.'

So I set off with my step ladder and canned air trying to be professional. When I reached the room the audience was mingling in the back getting teas and biscuits. I quietly set about removing the filters and dusting them out of a window. I then climbed my step ladder in the middle of the room under the projector and before replacing the filters I blew the inside of the projector with the canned air. BIG MISTAKE! The canned air meeting the hot bulb caused an implosion resulting in two fireballs and plenty of dust exiting each side of the projector in spectacular fashion if this was not enough to attract the attention of the room then the floating ceiling tiles being pushed into the roof void certainly was. As previously said this was a new building in 2010. The perfectly hung fire doors with their fluffy door edging meant the air from the blast had nowhere to go so pushed up the tiles.

This was by far the most embarrassing moment of my IT career. I turned to the onlookers and said 'Well that definitely cleared the dust' Nobody was impressed. I reset the dozen or more tiles. Reset the filter status and cleared off back to the IT department. Just thankful the projector works and remained fixed to the ceiling.

John (Sean) Begley

PGDip, CCNP, MCITP

[Madhan_kumar_G]

Ghostly Greetings!

Years ago, When I was an engineer handling a large university Wi-Fi, the horror happened. That was the time chrome books were launched in the market. All the students received the Chromebooks as gifts from the college. When all the students connected their chrome books to the Wi-Fi at a time, I had to confront my worst fears.

The Core N/W modules went down. People were screaming all over the place. Replaced modules went down consequently. That night went with fierce battle with the Capacity/Resource optimization & QoS. Finally, we won against all odds. But, following few days, my nights were filled with horror dreams.

Happy Haunting!

[CharlesJN]

About ten years ago, I was asked to design a network as my junior carrier at some place far from the city for a quite big school with around 100 staff and educators. 

At that time, all desktops were connected through LAN cable for intranet and internet access. Thus, I embraced the wireless connection by preparing all the necessities with a minimum budget and 1 Mbps internet speed.

[MerryAki]

My IT Horror Story is: How ridiculous it was to have >95% wired clients 😄 times have changed and WiFi accelerated that journey.

Real horror storys are: IT Security threats, stupid users and supply chain issues 🤣🤣

[JEdwards-Etex]

Seven years ago (On a night much like tonight)

In -someone's- first year managing Cisco routers, after only having a few months of Brocade exposure, -someone- learned how not to remove an interface from an OSPF area...  -someone- also learned that commit confirm exists in IOS-XR !

router ospf 100

 area 0

  no interface tengige0/0/0/1

   passive enable

end

y

The look of fear swept across their face (A face much like my own)

15 minutes later without remote management access -someone- realized that -someone- didn't delete "passive enable" from the syntax before -someone- pasted it in...well, time to go console in for management now!  Hopefully the network will survive until -someone- gets there.

[jrsilvius]

We once spent big money to move our company to a new and exciting web based, security appliance, and after many years of using this product, we invested more money in adding their Wireless Access Points and Switches. Then one afternoon said company issued a firmware update that has wreaked everything, and we have to allow unsecure traffic to pass our firewall so that we can get approved traffic. We've opened ticket after ticket and the company keeps telling us that the issue will be resolved soon, or you have to upgrade your production environment to the newest stable version so we can troubleshoot your problem. If only they had an allow feature for the traffic they keep blocking with as false positives.

[AmyReyes]

UPDATE: This contest has now ended. Thank you all for frightening us with your spooky stories! We have loved reading them. Congratulations to our randomly selected winners: @Polymathink, @EJN, and @Madhan_kumar_G. 🎉 

[alemabrahao]

Congrats guys.  🎉

[Inderdeep]

Congratulations @Polymathink, @EJN, and @Madhan_kumar_G. 

[Polymathink]

I'd like to thank the academy....Hey, is this thing on?