Dashboard Issues?

[Twitch]

Anyone else seeing weird issues with the Dashboard today? I was able to access it just fine earlier today, but then access to the it just went belly-up. Not long after that, I began receiving all of these notifications that devices are going down at all sites, but when I look at them in the Dashboard (which became reachable again) they appear to be just fine. The fact that my email and phone are not blowing-up with people in a panic seems to confirm this.

Are there any known-issues at the moment??

Thanks!

Twitch

[TimBisel]

I have gotten >70 alerts in 10 minutes for three organizations across three states and close to 40 networks. All other monitoring I have in place show nothing but alerts from Meraki are going nuts.

[Felix_moreno]

Same issue here. Waiting to get TAC on the phone so they can confirm. Last time we experienced this, the root cause was maintenance being done on the Meraki Network my company resides in. 

[vassallon]

@Twitch 

Yeah it appears that the Dashboard was not accessible for 10ish minutes. In that time frame devices were not able to check in which is triggering the email messages about devices being offline when in fact they have been working fine the whole time. 

[Dylan_YYC]

Yep, same thing here!

[Rayw]

I got kicked out of the dashboard and then couldnt get logged back in. Now I am getting a ton of email alerts about my VPN tunnels being down.

[AAJ]

We had the same experience. All our tunnels went down, but why ?  Meraki  always say that the network would still function without the dashboard, you just can't manage it.   Clearly that was not the case today.

[DHAnderson]

When you say the tunnels went down, did they actually go down, and no data could be transferred, or that the dashboard reported them down?

[RodrigoC]

Hey @AAJ,

Dashboard's only role in the formation of an Auto-VPN tunnel is sharing Port and IP info with Peer MXs. A Dashboard issue like this would make previously established tunnels show offline on the portal, but remain up in actuality. Were you unable to access resources on the other side of your tunnels?

[srbright]

@RodrigoC What would be the recommendation on determining the real tunnel status?  Or do we have to test each tunnel comm individually on our own, which is impractical.   Are we just blind until this is resolved?  

[RodrigoC]

Hey @srbright,

The best way to confirm if your tunnels are working would be to send some traffic across some of them. For some people, testing all tunnels might not be viable. 

My advice for anyone in this situation would be to test your most critical tunnels, confirm that traffic is flowing, and then send an FYI to people using those tunnels to let you know if they encounter any issues.

[nb12345]

Hello,

Did you confirm the RCA? what was the reason for outage please?

Thanks,

[Aztec_Ninja]

Not the best advice during this potential outage, but I learned my lesson some years back.  We stood up PRTG to monitor our network in additional to the Merkai alerts. 

I still almost wet my fresh pants when seeing all of the Meraki alerts.

[Twitch]

@Aztec_NinjaI know that feeling! We all need some Meraki-branded Depends if this is going to keep happening.

[lpopejoy]

"Meraki-branded Depends"  

...You know, they are big on swag...  😂

[Twitch]

@lpopejoy   😂🤣 

Could you imagine? Rock Your Depends Day at the office. Heaven help us...

[RodrigoC · Accepted solution]

Hey Everyone!

Checking in to provide a quick update. We are still working on putting together a full public RCA  to share with you all. Due to the nature of the issue, this might take several more days.

That said, I wanted to share what we know for certain at this time.

There was an outage on March 15th that affected customers hosted in the Americas region. The outage started 10:55 PT and lasted 8 minutes. During this time, some customers were not able to log into or access Dashboard. All services were restored at 11:03 PT.

The outage was due to a core network device managed by one of our data center service providers that needed to be rebooted and reconfigured. We are working with the provider to ensure this type of issue does not happen in the future.

Dashboard (dashboard.meraki.com) was unavailable during this time, however, there was no disruption to network connectivity and services at customer locations. Meraki devices were online throughout, though they will report as being offline during that time.

I hope this proves helpful and as always, please don't hesitate to reach out if there is anything else I can clarify for you.

Happy Tuesday to you all! 😃 

[ITDept-GT]

Is there a status page or email list that we can subscribe to, to be notified of these events?

[DHAnderson]

The Dashboard seems to be down again.

I am able to sign in, select a customer, but the page hangs or takes an extremely long time to display.

[rconiv]

A bit ago when I opened the dashboard, and it logged me in, it hung just showing the Meraki logo, and Health.  Nothing else was shown, though if I clicked on one of the news articles, it would show the full page.  After a bit though the issue went away.  I would select a users VPN and it works fine, though that might not be the same thing as what you are selecting.  Outside of the one page, everything else came up at their normal speed.

[DHAnderson]

It has come back up for me now.  It must be transient issues and it has happened a couple of times now.

[nb12345]

Seems to be working for me on the mobile meraki app? 

[BlakeRichardson]

@Aztec_Ninja  +1 for PRTG, wil its expensive its very good. We use that as one of our pieces of monitoring software. 

There is a free version with 100 sensors. 

[ValED]

Experienced the same thing, from 10:54 AM to 11AM.

[drgnslyr]

yeah, my NOC is pissed, 200+ tickets in 20 minutes, and no legit known issues.  Not sure what broke on Meraki's backend, but that is really annoying on a Monday.

[ZeuS41]

Same for us, all our VPN tunnels dropped.

[Aztec_Ninja]

Same here, getting all sorts of Network down alerts.  I just got a 2nd wave of alerts.  The dashboard looks good, though.

[DHAnderson]

Same here.  Email messages still pouring in.  I have clients in three different cities on several different providers.  I stopped panicking as no clients are calling me saying they are down.

It is interesting that the emails are in waves, switches, then access points, then cameras, finally firewalls. 

[arturbitt]

Same here!

[ccipnet]

We are having the same issues.  All of our Meraki networks' devices are losing connectivity (down/up) with the Meraki Cloud.  This started for us at 1:50pm EST and the last loss of connectivity was at 2:25pm EST.  We have had 4 cycles of down/up connectivity changes.

[ZeuS41]

looks like Meraki does have a lot of explaining to do, we bought into this "cloud" idea thinking the clients should not be impacted incase if the Meraki dashboard is unreachable, it turns out all VPN tunnels and clients access are linked directly to the cloud. the dashboard should only be used for management not access. if they dont fix this, we are switching our entire Meraki platform very very soon.

[DHAnderson]

My clients and tunnels are still up, otherwise I would have heard. They absolutely rely on Meraki, and not one of them have reported any issues. I even called on client and they reported everything was working.  All of my clients absolutely rely on Meraki, and not one of them have reported any issues.

My own equipment reported that it was down, but my home office kept humming along, with emails pouring in!

[Aztec_Ninja]

Did your site to site VPN go down?  

We have about  21 sites across the US and all our tunnels are up.  We use a separate monitoring tool to monitor.

It is alarming to receive all of the up/down alerts thought.  

[drgnslyr]

Reviewing our separate network monitoring, no Site to Site tunnel issues, all stayed up, we have 50+ locations.  Haven't heard any reports of outages of issues impacting end-users, thankfully.

[RodrigoC]

Hey Everyone,

Justa wanted to jump in and let everyone know that we are aware of the ongoing issues with Dashboard. As of now, we have confirmed that the issue is isolated to a small subset of servers, and as of 20ish minutes ago, service impact should be mitigated. We will share more information once we have conducted a full investigation into the issue.

If you are still experiencing issues, please feel free to let me know.

All the best,

Rodrigo

Meraki Support

[Twitch]

Hey @RodrigoC - Thanks very much for the update. We appreciate everything you guys are doing to figure-out what's going on.

[RodrigoC]

Hey @Twitch,

Thank you for your kind words. We are deeply sorry for the inconvenience and we'll keep you all appraised as things develop. In the meantime, if anyone has any questions or concerns, feel free to DM or tag me! 

[Twitch]

Our site-to-site VPNs have remained operational. They still show up/up and green. I haven't heard anything to the contrary in terms of remote site connectivity.

[Wayner]

I have had all of my Security appliances reporting they are going down but they are not. no other equipment has generated any emails. I have been on hold with support for 30 min so i am assuming it is a big issue for them

[Twitch]

Thanks, everyone. Good to know it's not just our network.

Other than the alerts, All is Quiet on the Western Front in terms of user complaints.

[Blueshift]

Can confirm here.
Not only was cloud unreachable, logs show all of our wireless devices re-authing with the APs and all of our manually disabled ports on the switches are now re-enabled.

No users seem to have been affected, however.

[ZeeBoussaid]

same goes for us, we have 200 remote office with VPN tunnels dropped. what happened today should scare the crap out of any engineer. this is going to be a nightmare if all access to clients is tied directly to the Meraki Dashboard. i hope Meraki fix this issue or at least chime in. because im seriously considering dropping all Meraki products in our environment.

[rconiv]

Getting it here also, had a recorded outage from 10:52 to 11:01 PDT at both our local and colo, and then been getting some emails since for either both locations or just the colo loosing/gaining connection to the Meraki Cloud.  As others have said, nothing seems to have really died because there was no screaming.  Have been getting occasional emails since mostly for the colo.  Am on hold with Meraki, who I am pretty sure we have overwhelmed them. 

[Remo]

This happened to my NOC.

Multiple sites seemed to have dropped for about 7 minutes, but there was one site that we manage that was not affected.

Meraki confirmed that this was only a cloud management outage as hosted services migrated to another site since the affected site was under an outage. No actual service impacted issues, just management/visibility.

I imagine you'll get the full details if you call them or open a ticket with them against one of the networks that were affected in your organization.

[PDSKturley]

For what it's worth, None of our Meraki VPN's or site to site VPN's actually dropped. Not a singe call from a user about disconnects or WiFi access lost.

While this is a serious issue, we are relieved it didn't actually affect usability. Just the management portal. Meraki will certainly have to answer the questions about why. And reassure us this platform is resilient. I would expect an explanation of the root cause, and details about the resolution for future mitigation should it happen again.

[Twitch]

Same here - we never lost functionality in terms of user connectivity or connectivity between sites, there was just a ton of emails saying that nothing was reachable and everything was going down all at once at all sites.

It was clearly enough to create an "Oh Sh*t" moment, and caused me to start remembering all of the changes I made earlier in the day.

You know the moment: "Dammit. Did I cause this?????" While sprinting away from that half-made cup of coffee back to your computer at full speed...

[DHAnderson]

I had just cooking hot dogs for lunch when my phone lit up.  My poor hotdog boiled to death while I was pouring over email and making calls to clients.

[Twitch]

@DHAnderson  Oh man, that's hilarious. RIP to your wiener. What a way to go...

🤣🤣🤣

[RodrigoC]

Hey @DHAnderson,

Sorry to hear about your hot dogs 😞

If you are ever in Chicago (post-COVID) near the Meraki Offices, I'd be happy to treat you to a Chicago Dog!

[ZeeBoussaid]

i can confirm what others said about the VPN drop. i was looking at the wrong logs, all VPN tunnels remained operational. but still Meraki has to mitigate the issue

[tantony]

Yes, I had dashboard issue for about 5 minutes, around 2 PM eastern time.

[Bearb]

Our Orgs received not only alerts for networks with VPN, but all devices in those networks with VPN produced alerts in Meraki.  We also saw Meraki alerts for our other networks without VPN or Meraki firewalls (MR's).  So this was not focused on just VPN networks.  We also monitor some of our Meraki gear with PRTG as well as snmp.meraki.com which showed down for about 8 minutes during the issues.  

Thankfully I'm not aware of any of our networks actually experiencing issues during this outage.  Looking forward to reading the RCA, and now Azure looks to want to join in on the Monday fun with their own issues.  Happy Monday everyone!

[rconiv]

Just saw that Twitter alert a bit ago about M365 issues.  I can't get in to the admin portal right now, either give a not available, or only loads a bit of it.

(Edit) Though now it works.  That was pretty quick for them.

[DHAnderson]

O365 was down due to Azure Active Directory having issues.  That also brought Cisco Duo to send me this email subject line:

Azure Conditional Access Authentication failures.

It is truly the ides of March!