[CONTEST CLOSED] Haunt us with your IT horror stories! Win a treat!

[AmyReyes]

UPDATE: This contest has ended. Thank you all for frightening us with your spooky stories! Our randomly selected winners are: @TMRoberts, @van604, and @DwightHirst. 🎉 Congratulations!

October is here, which means Halloween is just around the corner. Who doesn’t love an excuse to dress up and eat candy?! We don’t have any tricks in store for you this month, but there just may be some treats… read on for details of how you can secure your chance to win a Meraki Nike Dri-Fit Hat. 

Spooky season is the perfect time to give each other a fright, and what better way to do that than by sharing your IT horror stories? We thought it would be fun to bring back this hauntingly good Halloween contest for another year. 

So gather round, pull up a virtual chair by the light of our jack o’lantern, and let’s tell some ghost stories! We want to hear your IT horror stories… Regale us with your tales of nightmarish networks, spooky servers, and petrifying permissions. What’s the scariest scenario you’ve encountered? Remember, picture submissions count, too! 

How to win

To enter the contest, share your IT horror story or picture in the comments below before 10:59am ET on October 13th. Be careful not to use any company names, and be kind in your tales. We will then randomly draw 3 winners to receive a prize.

Read on, if you dare...

If you’re in need of a little inspiration, we chose a few fan favorites from last year’s contest to get you started. 

First up, this truly terrifying photo entry from @Hezinuk:

Customer: We think we have an issue with our switch! 

Me: Let me have a look

Or this real life ghost story from @TMRoberts:

“Oh I had an interesting and spooky time at the one place .... worked with security guards who complained they thought someone had broken in while they were on duty.

So, we ended up going through camera footage of the nights they said they thought it happened. 

You see the guard walk down some stairwell (internal so no wind), and as they walk past the one door it pops open and closed .... no one on either side. Dude freaks and rattles handle ... maglocked.

Then he opens door peered through, still no one there.

Walks away shaking head .. gets near the next floor... that door pops wide open (not like a little magdoor pop, WIDE open) ...

Sec guard runs down last steps and looks through, still no one there.

We went through all the cameras at the doors and nearby hallways ... not a soul was in the building .... except this guard.

Next day, different guard mentioned earlier in week they had been in a stall in restroom ... door opened .... again no one else in building when we checked the cameras.

Glad we moved buildings two months later!!”

oOoOo scary! Happy Halloween everyone! 👻🎃🍬

TERMS & CONDITIONS

• Limit one entry per community member
• Submission period: Monday, October 2nd, 2023  at 11:00am EDT through Friday, October 13th, 2023 at 10:59am EDT
• Prize will be a selection of Meraki swag with value not exceeding USD 50.00
• Official terms, conditions, and eligibility information

[van604]

got called to a remote site, 2 hour flight that network was down. for 1/2 the office.  Flew in with a some gear, new Meraki switch box of cat6 and tools.  Walked into this joy of a dead switch sitting under a leaking toilet, glad i brought a few pairs of disposable gloves.  so long HP hello Meraki. 

[Jagatiya143]

I believe picture will tell the story. Before I deployed Cisco Meraki!!!!🤐

Before I deployed Cisco Meraki !!!

After I deploy Cisco Meraki !!!! 🎉🏆 (Can not shar ethe actual pictures)

[DarrenOC]

Too many stories to share but this one always springs to mind:

many years ago as a junior field engineer with some IP telephony experience I was asked to upgrade the IOS of two voice gateways in the US (I live in the UK).  Not an issue, nice and easy task to upgrade the IOS on Cisco routers.

Did the first upgrade, router came back up successfully and I still had connectivity to the site. Down went the next router and with it the whole site. 15 minutes later still nothing.  Many escalations later to our senior support team I was told the routers were running BGP and the IOS i had installed didn’t have that feature set.

That was the day I’d taken down a very well known airline in Norwalk.  I was mortified and horrified at what I’d done but I very quickly learnt the difference in the IOS feature sets.

[BlakeRichardson]

This is an issue I dealt with earlier this year. an external comms cabinet that some ants decided to call home. Needless to say the switch remained fully operational while being home to numerous residents.

After a good clean out the switch still lives on.

[RARAO_PROINF]

The worst cabling ever seen

[SimoneP]

One cold day many years ago, I received a call and answered.

Me: Yes?
the other party: frrrufhf (undefined noises)
me: who is this? can't hear you?
The other party: frrufhfhf (undefined noises).

I decided to end the call.

The fog outside the window became thicker and the phone rang again:
Me: yes?
the other party: frrrufhfhf (undefined noises), we are company xx, we need a network technician, can you come in? we are hung up, the network keeps going down on Wednesdays. As if there were ghosts and then it starts again.
Me: I asked for various indications and details, this company was new to us and we couldn't intervene without indications
The other party gave us all the necessary directions and off we went.

We arrived like the ghostbusters in the film, ready to catch these ghosts. It was cold that day, there was a lot of fog.

The company was small and the rack was horrifying. Unmarked cables, randomly placed, no rules.

We tried to investigate and despite all the mismanagement, the problem only occurred for about 10 minutes on Wednesday mornings.

We decided to return the following Wednesday, analysing the network and trying to understand.

Suddenly, a clap of thunder jolted us, it was raining hard outside, but the net still held. Suddenly: total darkness. The net was down.

We immediately went to the rack room and found the ghost of Wednesday: the cleaning lady. Yes, that's her.

Every Wednesday morning he cleaned the offices and broke into the rack. He would unplug the first socket of the power strip to plug in his hoover. The first socket obviously powered the switch.

After a while, there was an overall reorganisation of the rack room, with new meraki machines and obviously also a rack key, to prevent any 'ghosts' from opening and unplugging anything.

[MK2]

It is pictures like these, here just a small IDF, that trigger the horror in me and make me deeply sad on the other side. When the customer says I have VLANs, but it looks more like PLANs (physical...) :D.
I said: The coding with the cable colors is yes well solved.
Customer: Coding? Um, those were the cables we had at hand.
Me: Uffff....
And so on :).

I hope one or the other of you knows something like that 😉

[donLupp]

Spoiler

Meraki from Hell !

😈👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿👿

Meraki from Hell

[kydave]

I used to work in a large hospital. One of the IDF closets in the basement was located inside another room. That's the door to the IDF, right behind that big box-shaped cart. So when you went in the IDF to work on something, you never knew when someone would push a cart back in and block the door shut. Oh yeah -- that box-shaped cart is the morgue cart.

The real problem with that IDF room was that the entire outer room could get filled with lots of carts, beds, etc, without warning -- to the point that it was impossible to get the IDF door open at all. I had a friend who went in the IDF to work for a while, and the door closed. A short time later, hospital staff filled up the outer room completely. He was by himself, with no one anywhere near, so he had to use the two-way radio he brought with him to call for help.

That's the IDF door toward the back, impossible to get in or out of. And yes, that's the morgue cart again, over against the opposite wall. Complete scary log jam.

[TMRoberts]

My old story was definitely spooky but as for horror.... a previous place I had just started had 1/2 on WAN, half on an outdated Citrix mini-farm. 

So besides Citrix there, the main horror was if and only if a computer had Anti-Malware it was an on premise updating only old setup .... wouldn't download from the internet for clients especially if they left the WAN. Day of installing a newer cloud-based SaaS version of a new vendor .... someone clicked on a mallard loaded PDF in Citrix presented Outlook ... which was also attached to our file server!

Needless to say it was also ransomware too ... so the payload ran on the Citrix session and ran all throughout there, it saw the connected file server and started attacking that too. THANKFULLY I had just installed the newer Anti-Malware suite on the file server and it was doing updates as the user opened the ransomware .... I had installed on DCs earlier and was going to tackle the file server then Citrix. End users PCs we had pushed from LMI earlier so we're covered as they came online.... 

Now since we had also updated branches to Meraki MXs or Z series ... we had a newer functional WAN and I got a quote to have to migrate our Citrix environment (legacy needed 3rd party assistance per Citrix) or replace remaining branches with MX/Z series .... that was lower and in stock so we ordered them, had them shipped to branches and had complete WAN and I just shut off Citrix and called it a day!!! Horror story but fixed with Meraki!

[TMRoberts]

@AmyReyes .... so about the dry fit ... I know Cisco has a merchstore ... is there a Merkai one yet for fans??

[KarstenI]

My story, which happened already about 25 years ago, is a little bit similar to the one from @BlakeRichardson:

On two different occasions the customers complained about a strange smell in the office. Well, the first time it was a dead mouse inside the desktop PC, the second time a dead mouse in the bottom pontoon. While the PC was cleaned, they decided to fill the bottom pontoon with construction foam without getting the dead mous out of it ... Sadly (or luckily?) it was not common those days to carry a camara with you all the time.

[KarstenI]

@TMRoberts Not a store, but there are fan articles: https://ciscomeraki.influitive.com.

[KarstenI]

@DarrenOC started with personal failures. I also have to contribute two of mine (although there are more):

1. A long time ago wanted to migrate the single drive Novell Netware server to a RAID system. In the middle of the process I realized that I mirrored the new and empty disk onto the production disk. Luckily the backup was up to date and the restore went as expected.
2. A new customer where I was looking for a place for the new firewall in the crowded rack. I touched a very tight ethernet cable and suddenly the noise in the rack got reduced. What I didn't see was that the cable was crossing the power-switch of the multi-outlet power strip on the other side of the rack. And this power strip connected both controllers of the SAN.

[DarrenOC]

@KarstenI You learn an awful very quickly from failure. Doesn’t feel good at the time but you bounce back stronger and tougher

[AmyReyes]

@TMRoberts - I know you mentioned the Cisco Store... not sure if you've seen it, but there is a Meraki section of swag in there at https://merchandise.cisco.com/featured/cisco-meraki.html 😃 

We're going to be ordering more prizes for the Community in the next couple of months from our vendor, too, so if you have any special requests, let me know! I'll see what we can do. 

[cmr]

In our main datacentre we had a flash array used as the primary storage that kept having weird issues with the four drives that contained the system partitions. 

One day we had a battery backup fail on one of the controller cards so an engineer came in from said vendor to replace this part.  Unfortunately during the process one of the system partitions dropped offline completely leaving us with an operational, but not highly available array.  Despire trying multiple spare flash drives we had on site, the engineer could not get it back online, so as it was Friday night and the weekend is our busy time so we went home while the engineer wrote up his notes before deciding what to no next week.

For some mad reason he was allowed back into the room by security and decided to test one more thing, by putting the original flash drive back into the array and crashed the whole thing, taking down our EMEA operations at 11pm on a Friday night!

The vendor then offered no solutions to remedy the situation so after a mad 3 hours to get the business back online and a week or so of our work to recover the remaining corrupted data, we purchased a mirrored pair of replacement arrays from a different vendor...

It still makes me stressed thinking about it!

[DwightHirst]

We acquired a new business in late 2022 and were tasked with decommissioning their IT infrastructure and bringing it all back to the site. After a 3-hour drive, we were greeted by their main server cab at the entrance to their public storefront with only a clear front cab door to protect it from the outside world. The cherry on the top was that the door didn’t lock due to the housing being rusted and a breeze from the doorway could swing the cab door open, an open invitation to anyone with even the slightest inclination towards mischief. It was a security nightmare.

[tmayes0287]

In the heart of a remote and desolate landscape, I found myself at an abandoned research station, nestled far from civilization. The only connection to the outside world was an ancient ISR 4321 router, which provided a meager 1.5 x 1.5 Mbps of internet speed to the few remaining inhabitants of the station. The isolation was palpable, the winds howling like tormented spirits as I ventured to upgrade the archaic system to the advanced MX68CW, promising a miraculous 100 x 100 Mbps of internet speed.

The journey to the research station was harrowing, taking me through miles of barren wilderness. As I approached, a sense of foreboding crept over me. The station stood like a haunted relic against the backdrop of an endless, snow-covered tundra.

The old ISR 4321 seemed to exude a weary and forlorn aura. Its fans whirred with an eerie, almost mournful hum as I unplugged the cables, disconnecting it from the world it had known for so long. It was as if the router had absorbed the solitude of the station, and it resisted its replacement with a quiet resistance.

I mounted the MX68CW, and as it powered on, the room seemed to shake, and a chilling wind swept through the decrepit station. I brushed it off as my imagination running wild in the lonely wilderness. The router configuration went smoothly, and I marveled at the promise of fast internet that was within reach.

But just as I was about to celebrate my success, a spine-tingling whisper echoed through the desolate station. I dismissed it as the wind, but it grew louder, and I realized the station was not as abandoned as it seemed.

The walls of the station began to creak and groan, as if awakening from a long slumber. Ghostly figures materialized out of the shadows, their faces twisted in anguish. These were the lost souls of the previous researchers, trapped in the icy isolation, yearning for human connection.

Their ghastly presence intensified, and they approached me with outstretched, frozen hands. Their eyes bore into my soul, and I felt an icy grip tightening around my heart. Desperation and isolation had turned them into vengeful spirits.

I frantically attempted to complete the installation, hoping that the MX68CW's promise of high-speed internet would somehow save me. As the router fully powered on, the spirits let out a collective, agonized wail. The lights flickered, and with one final surge, the router unleashed its promised 100 x 100 Mbps of internet speed.

In an instant, the spirits were engulfed by the blinding light of the internet, and with a collective scream, they vanished, leaving only the echoes of their torment behind.

I knew that I had brought the only relief these lost souls would ever find - a fleeting connection to the outside world. As I left the forsaken station behind, I couldn't shake the feeling that the ghosts of the past would forever linger in the dark corners of the digital realm, connected but eternally trapped in their icy isolation.

[AmyReyes]

UPDATE: This contest has ended. Thank you all for frightening us with your spooky stories! Our randomly selected winners are: @TMRoberts, @van604, and @DwightHirst. 🎉 Congratulations!

[Inderdeep]

@TMRoberts, @van604, and @DwightHirst. Congratulations!

[Brash]

Thanks for all the stories everyone, they were a great read.

And congratulations @TMRoberts , @van604 and @DwightHirst 

[BlakeRichardson]

Well done everyone and congrats to the winners. Some great stories there!