[WINNERS ANNOUNCED] Community Challenge: The Impossible Fix

[MeredithW]

UPDATE: This contest has ended. Huge thanks to everyone for sharing your stories, we loved reading all of them! Congratulations to our three randomly selected winners: @jfry2k, @MakkyCro, and @LuisCruz.   

For many of us, it’s been a few months since our sudden transition to the world of remote work.  We’ve had to navigate more than a few snags in our setups, such as unexplained internet outages, network security breaches, and even hardware malfunctions (maybe your keyboard and the contents of your Diet Coke became close friends). All requiring contact-free intervention.

For this month’s challenge, we want to know about the craziest / most  painful / most impossible fix you’ve had to resolve remotely using Meraki. (This could be a fix made during or prior to COVID)

Here is an array of colorful examples provided by my colleague @Phi-L, Technical Trainer to the Stars:

1. Maybe an intern decided to plug in all the loose cabling they could find in a room back into a switch, introducing a loop. You used the dashboard to troubleshoot RSTP issues.
2. You’re the dad of a wallowing son, who now is stuck back at home due to COVID closing his college. In his despair, he is sucking up all the bandwidth by streaming Twitch and playing PS4 all day. You used traffic shaping rules to deprioritize that traffic and handed him a copy of One Hundred Years of Solitude to fill his time.
3. Perhaps someone was INSISTENT that a device on a wall was an AP, but you used the blink LEDs live tool to track down the AP he was actually looking for and prove the mystery device to be a smoke detector.

How to enter

Post a comment on this blog post containing your story about an impossible fix that you’ve pulled off remotely. Submit your comment before June 23rd at 11am PT and be entered to win a fantastic specimen of Meraki swag (a Meraki mini block set!):

We’ll then randomly select three winners from the list of entrants. Good luck!!

Fine Print:

• The winners will be announced on Tuesday, June 23rd at 11am PDT
• One entry per community member
• Full terms, conditions, and eligibility information

[RupertWever]

Meraki dashboard did not allow me to change STP root;  Meraki support team was professional, yet unable to perform the task either.  I happen to be working on python scripts to automate other tasks so I built a quick postman call to make the change: success!  I believe case is still open for Meraki development to investigate.

[jfry2k]

Two words... 

Firmware Bugs.

[RupertWever]

Meraki dashboard only allows download of network event-logs for one page at a time, and we needed longer history list of events.  So I built a python script that downloads 500 pages worth of event logs history.  Now I'm even adding additional lookups to include client vendor so we can look for patterns.

[ebetancourt]

Mission 1 question 3 is broken.

How many daily API requests are made through the Meraki dashboard?

I did put 45million and it marked it as wrong.

[OswaldoEcuador]

My first installation we dont now how to begin the installation and only with the markai pageI did it

[TroyW]

I had a customer that had a MX fail and would not respond.  Luckly the customer had and old MX on site and was able to swap the units and I was able to temporarily get them up and running.  Then I did an rma of the failed unit.  When the customer received the replacement, I was able to put it back into production, with out ever going on site.

[Stealth_Network]

We had a customer with a problem on a VPN between the US and a Canada data center. The VPN was up but traffic was dead slow through it. I used MTR to discover the issue with between two ISP's at the border, the handoff was introducing huge latency between the two. I was able to re-route traffic to another Canadian site that had a VPN back to the main Canadian hub using the Exit Hub configuration. This worked until the ISP issues were resolved a week later.

This branch would have been essentially down without this option.

MTR was valuable in showing both ISP's (you know the finger pointing game) where the issue was.

Thanks 

[Mikanator]

Migrating the corporate wide wireless authentication and authorization infrastructure to a new ISE implementation is something that causes MANY eyes to be watching and looking.

Meraki's built in tools and packet capture capabilities allow the testing, and validation and POC's to be all done REMOTELY.

We were up against a hard deadline.

Then the current situation hit.

The tools in Meraki's platform allow the implementation, POST validation and POST monitoring and validation to occur without a hitch.

A few incidents tickets came in but the tools in the platform allow us to see that the issues were because the users were using improper wifi setup on the devices or bad credentials.

Work smarter not harder.

Thanks Meraki for letting this happen!

[Network-dad]

We were having issues passing PCI scans due to Meraki Client VPN.. After several call Meraki Support change the Client VPN Encryption to more stringent requirements (AES128 encryption with DH group 14 - Required by PCI-DSS 3.2). Now had to update all my end points to use this new encryption standard and configure the Meraki Client VPN... I was able to cerate a small PowerShell Script to automate this process for me and set all the settings needed.

$ServerAddress = "xxx.xxx.xxx.xxx"
$ConnectionName = "Meraki Client VPN"
$PresharedKey = "A Password"
Add-VpnConnection -Name "$ConnectionName" -ServerAddress "$ServerAddress" -TunnelType L2tp -L2tpPsk "$PresharedKey" -AuthenticationMethod PAP -Force
Start-Sleep -m 100
New-NetIPsecMainModeCryptoProposal -Encryption AESGCM128 -Hash SHA1 -KeyExchange DH14

[MakkyCro]

Got a call during holiday that the WiFi was down.
With the help of the dashboard, and the Meraki support team, we identified a firmware bug.

The happy ending of the story, even being remote, I was able to restore the WiFi only using the dashboard 🙂
Bad side of the story, ended up using a beta firmware on a production network.

[LuisCruz]

At some point the number of organizations in my dashboard, including test orgs, was over 20 so I decided to do some clean up, so I started deleting test organizations and accidentally deleted a client semi-production/testing network. Wooow! I almost die, but was able to recover 95% of the config just by following step by step the Dashboard Change-log. The client was cool about it, he just asked me "are we back up?", I said "Of course we are! this is Meraki 🙂

[Andrewbluepiano]

My parents home runs Meraki. One day my mother called me because their internet had gone down. After a bit of ranting and raving about how it doesn't make sense for me to run their house on enterprise equipment when I cant be home to manage it, my mother said she would ask my older brother who lives close to them to check it out later in the day. Given that his expertise is IT management, and a lack of a login for their network, that was going to be tough. 

I checked their network, and saw the issue was just a cable that had been unplugged, which was strange as nobody goes near the networking stuff besides me. Texted my father and brother who would be at the house later in the day to let them know which cable it was so one of them could fix it.

Turns out my brother had been walking around the basement earlier that day and stepped on the cabling. 🤦‍

[BAllen]

be me

systems engineer at a small computer consulting company

client calls complaining they cant use vpn

try to access vpn

doesn't work because i run linux

client isn't far away

get in 2015 mustang ecoboost to speed on over

client uses shared office space

go into lobby

badge doesnt work

go to security desk

"uuuuh my badge doesnt work"

cant go in without an appointment because covid

explain that the vpn is down

still cant go in

go back to the 'stang

get an epic idea

drive around the building searching for signal

see my clients ssid thanks to the wall penetrating power of the Meraki MR33 2.4Ghz Channel

connect to network

ssh to clients asa

fix vpn

Spoiler

save the day

[Bille]

Migrating a client to a new ISP. 

I was able to help a client move to a new ISP fully remotely. They received all the new IP information in advance, and I was able to via VPN switch port 2 on an MX to a wan port and configure it with the new IP information. Further more, I configured new 1:1 NAT rules in advance in the dashboard and advised the client to prepare their DNS with lower TTL values to speed up the change over to new public IP addresses at the time of the cutover. The client was then able to plug in the new ISPs connection in port to, and we were able to set WAN 2 as primary uplink to migrate the client over to the new ISP.

The client is based in Sweden and I did this whilst locked down in my flat in the UK. 

[Jwiley78]

Trying to make Meraki client VPN pass PCI scans.  5 calls with Meraki support and 4 calls with AT&T support.

Oh wait, problem still not resolved......

TO BE CONTINUED!

[Network-dad]

@Jwiley78  our Meraki Client VPN passes PCI scans.  You have to have support up your encryption to AESGCM128 and KeyExchange to DH14 then change the encryption level on your end points... I did this via PowerShell 

New-NetIPsecMainModeCryptoProposal -Encryption AESGCM128 -Hash SHA1 -KeyExchange DH14

[Jwiley78]

That has been my struggle.  Changing the end points.  Thanks for the script.  I'll try that.

Any luck with Macs?

[Network-dad]

we are a pure Windows environment so I've only tried it on windows.. here is my full script I use for deployment.. I'm working on a full write up on this as we speak.

$ServerAddress = "xxx.xxx.xxx.xxx"
$ConnectionName = "Meraki Client VPN"
$PresharedKey = "A Password"
Add-VpnConnection -Name "$ConnectionName" -ServerAddress "$ServerAddress" -TunnelType L2tp -L2tpPsk "$PresharedKey" -AuthenticationMethod PAP -Force
Start-Sleep -m 100
New-NetIPsecMainModeCryptoProposal -Encryption AESGCM128 -Hash SHA1 -KeyExchange DH14

[Jwiley78]

Nice, bookmarking this for later.  I've now got a project for tomorrow.  Hopefully it can be the impossible fix that is solved by the community.  🙂

[danhosking]

300 WAPs in 600 seconds (60 would have been better but this will do).

Due to the complexity of a building site and delays by subcontractors, I decided to licence and configure the entire wireless on my mobile while watching TV in under 10 minutes.

[Sheeraz]

I was doing R&D to create multiple networks at a time on the Meraki dashboard for a particular type of device, then I found a very quick and easy way to do it within the Meraki Dashboard by uploading the excel sheet with necessary information. In the end, I completed the task quickly.

[YKWong]

Meraki dashboard does not have notification when I had 2 WAN port activate and one of the WAN port is down. I need to manually login to meraki dashboard to check the wan port status.

[Zeolite]

One in a hundred meraki access points is unable to find a gateway to the internet. In the wireless - monitor - access points page says, 1 access point is repeater. Upon investigation, we find out that the cable is shorted. Thanks to Meraki Switch cable test we can check the cable even from home. 

[BlakeRichardson]

We had 10 iPads stolen about a year ago, I quickly setup a STOLEN tag and settings group that was deployed to all of the devices, that way if any of them came back online they would be bricked and display a nice wallpaper that clearly showed the device was stolen. The devices were registered in DEP so there was no way around this without jailbreaking the device. I also setup an alert that reported when the devices came online so I could log IP address information that was passed onto the police.

The end result was unearthing a large scale stolen goods and drugs ring.

Of the 10 devices 1 was recovered. 

[AhmedElsayyed]

One of the most challenging aspects of managing large distributed networks is troubleshooting issues when the client is across town (or maybe even across the country!). Having on-site IT personnel 24/7 at even small satellite branch offices can require a very large IT staff and is too expensive for most organizations. Meraki networks offer a variety of “remote hands” troubleshooting tools, helping network admins diagnose and resolve many wireless connectivity issues without dispatching IT staff to the site. The ability to run diagnostic checks such as pinging an access point, running a throughput test from Dashboard, or reviewing detailed event logs have been integral to Meraki’s value..

[Angus]

280 sites all with different ISP under different customer accounts and they are missing critical. Running blind if a site goes down with no insight or understanding of situation.

280 Z3s all VPN together with backup dongles. No more down time!!

[Ujjawal23]

2200+ device upgrade in just matter of few clicks. Wow moment for an engineer.  

[RupertWever]

I crossed the pond for vacation and happened to be in an Amsterdam tram when I got an email from a customer that their users where associating to the WiFi but not able to browse the internet. So, logged into the dashboard from my phone and helped troubleshoot (dns server issues) just before my Amsterdam-central station stop. #meraki-simplicity. 

[Merakian]

Situation: Due to covid-19 we have about 500 employees (throughout 13 western US states) working remotely using Virtual Desktops. 

Problem: We had a few days of VPN failures from Azure to our old Colo ASR (in Phnx, AZ) and our remote employees couldn't access critical network resources.

Resolution: Got a Meraki vMX100 trial license, and within 2hrs we had all VPN clients working again. Meraki vMX100 autovpn to our Colo MX400.

We decided to retire our old Cisco ASR and keep the Meraki vMX100.

All this magic from the comfort of my couch at home. Powerful technology that makes life simple!

[vaguirre17]

Fix issues with the startup ip addressing! Sometimes we have al least on MR as a bridge after try to fix the ip address we just lost forever the connection to the MR. 

Once I have all my AP as bridge but the switch wasn't registered. We had to go to the site and reconfigure the MS. 

[Guillaume6hat]

I needed to test third party VPN to DataCenter ASA from a MX remote site, as workaround solution.
As the encryption domain for 3rd Party VPN was the same as the MX in DC, I was stuck, it was not possible to do it on the same organization without impacting all sites.

Another idea came to me.

I had a MX cluster for tests purpose, with MS behind, and a computer. But as it was the Covid lockdown, I was not able to do any change physically.
I first disabled and changed the Vlan Id on the Switches for ports connected to the MX2, then removed MX2 from network and unclaimed it from Prod Organization. Then claimed it and added it on another network, in a Preprod Organization.
MX2 came back up in seconds, with new config, new subnet, and configured 3rd party VPN was UP.
I was then able to change the switch port connected to my remote computer to the new Vlan Id, and activate the MX2 LAN ports. I found the DHCP IP of the computer in the Dashboard, and I did all the tests I wanted with VNC.

As tests were successful, I then switched back to the normal state, disabled LAN ports and changed Vlan Id, removed and unclaimed MX2 on Preprod Organization, claimed it on Prod Organization, added as Warm Spare, activated LAN ports, switched back Vlan Id for the computer.
Again, my cluster was 100% functional in a few seconds.

From a MX Cluster, to 2 differents MX, in different networks, in different Organizations, testing on a computer, and then back to a MX Cluster, without any physical intervention, that was awesome ! 😁

[Vinaykanthala]

we (Meraki TAC) have completed Meraki Integration with Azure and tested with user domain credentials it’s working fine. checked with lobby admin credentials as well and tested the access dashboard with domain credentials.

[TMRoberts]

Hey, well, we had a fun one .... we had a 'mobile response trailer' at a job site and the owners wanted a way to view the site 'at their discretion' to see how it was progressing. Company does environmental cleanup and they had a tanker roll over and dump 10,000 gallons of fuel oil into wetlands that was going to take months to clean up.

They also wanted a way to call them there without tying up their cell phones and also have a printer on site anyone could use that showed up from the company .....

So Meraki Z3C to the rescue paired with a Webex Teams phone.

They wouldn't spring for a Meraki camera, but did go with a more traditional DVR that had DDNS connection, so we used the Meraki Z3C to get them on the same network and connected ..... installed the printer on the server on prem at the main branch, shared it out. The Z3 was technically on the WAN via it's cellular connection (mobile response trailer was literally mobile at times), printer was static IPd on the Z3 LAN, but still using print share from server, anyone could map the printer and send jobs. The DDNS DVR camera was plugged in and users could connect via their cell phone apps and then Webex Teams just needs PoE .... Z3C has one PoE port, which powered the phone and then users just used wifi from laptops to get to company data. Yes cell phone plan was high and we had overages, but the whole project was billable for equipment so SCORE ..... they needed wifi, cameras, phone etc ... Thanks Cisco for the great product suites!

[PeterJames]

We had one customer that logged quite a number of connectivity issues over a number of months. After several prior site visits and the customer using their own engineer to try to work it out, I put on my Meraki hat (ok, no hat!) and requested one of our senior engineers to attend site with me remote.

Working together we mapped out the estate and found a dozen more Meraki AP's that I had not listed. It turns out the customer installed their own Meraki AP's for customer use (ours was for EPoS) and in the logs I could see they were trying to work on the same channels and blacklist us. Ok, instead of going all cowboy duel on them (we would have won!) we agreed separate channels and to not block each other (I may have turned on blacklisting all their AP's for amusement). Interesting how they just assumed they needed to keep things separate for legislation (DSS PCI) purposes.

You would think this was the end of it, nope. We spoke to the staff on site to find out where the problem hot spots were and it was in a new area they opened up six months ago! A "hard" hmmm ensued and so did a few calls to certain people; "Why was I getting slack for a NEW EXPANSION?!"....anyway, long story short we re-did our WiFi survey and advised what was required.

Long story short...sometimes you just need a little more communication.

[vassallon]

My impossible fix is Meraki breaking how profiles are managed on iPads. 😤

[MeredithW]

This contest has now ended! Congratulations to our three randomly selected winners: @jfry2k, @MakkyCro, and @LuisCruz. Great stories everyone, thanks for sharing with us!

[Network-dad]

Congrats Guys!!!!! @jfry2k @MakkyCro  and @LuisCruz