cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Dashboard Issues?

SOLVED
Twitch
Getting noticed

Dashboard Issues?

Anyone else seeing weird issues with the Dashboard today? I was able to access it just fine earlier today, but then access to the it just went belly-up. Not long after that, I began receiving all of these notifications that devices are going down at all sites, but when I look at them in the Dashboard (which became reachable again) they appear to be just fine. The fact that my email and phone are not blowing-up with people in a panic seems to confirm this.

 

Are there any known-issues at the moment??

 

Thanks!

 

Twitch

 

 

1 ACCEPTED SOLUTION

Accepted Solutions
RodrigoC
Meraki Employee

Re: Dashboard Issues?

Hey Everyone!

 

Checking in to provide a quick update. We are still working on putting together a full public RCA  to share with you all. Due to the nature of the issue, this might take several more days.

 

That said, I wanted to share what we know for certain at this time.

 

There was an outage on March 15th that affected customers hosted in the Americas region. The outage started 10:55 PT and lasted 8 minutes. During this time, some customers were not able to log into or access Dashboard. All services were restored at 11:03 PT.

The outage was due to a core network device managed by one of our data center service providers that needed to be rebooted and reconfigured. We are working with the provider to ensure this type of issue does not happen in the future.

Dashboard (dashboard.meraki.com) was unavailable during this time, however, there was no disruption to network connectivity and services at customer locations. Meraki devices were online throughout, though they will report as being offline during that time.

 

I hope this proves helpful and as always, please don't hesitate to reach out if there is anything else I can clarify for you.

 

Happy Tuesday to you all! 😃 

View solution in original post

49 REPLIES 49
TimBisel
Getting noticed

Re: Dashboard Issues?

I have gotten >70 alerts in 10 minutes for three organizations across three states and close to 40 networks. All other monitoring I have in place show nothing but alerts from Meraki are going nuts.

Felix_moreno
Conversationalist

Re: Dashboard Issues?

Same issue here. Waiting to get TAC on the phone so they can confirm. Last time we experienced this, the root cause was maintenance being done on the Meraki Network my company resides in. 

Dylan_YYC
Getting noticed

Re: Dashboard Issues?

Yep, same thing here!

 

Rayw
New here

Re: Dashboard Issues?

I got kicked out of the dashboard and then couldnt get logged back in. Now I am getting a ton of email alerts about my VPN tunnels being down.

ValED
New here

Re: Dashboard Issues?

Experienced the same thing, from 10:54 AM to 11AM.

vassallon
Head in the Cloud

Re: Dashboard Issues?

@Twitch 

 

Yeah it appears that the Dashboard was not accessible for 10ish minutes. In that time frame devices were not able to check in which is triggering the email messages about devices being offline when in fact they have been working fine the whole time. 

Found this helpful? Give me some Kudos! (click on the little up-arrow below)
drgnslyr
Getting noticed

Re: Dashboard Issues?

yeah, my NOC is pissed, 200+ tickets in 20 minutes, and no legit known issues.  Not sure what broke on Meraki's backend, but that is really annoying on a Monday.

ZeuS41
Conversationalist

Re: Dashboard Issues?

Same for us, all our VPN tunnels dropped.

Aztec_Ninja
Here to help

Re: Dashboard Issues?

Same here, getting all sorts of Network down alerts.  I just got a 2nd wave of alerts.  The dashboard looks good, though.

arturbitt
Conversationalist

Re: Dashboard Issues?

Same here!

ccipnet
New here

Re: Dashboard Issues?

We are having the same issues.  All of our Meraki networks' devices are losing connectivity (down/up) with the Meraki Cloud.  This started for us at 1:50pm EST and the last loss of connectivity was at 2:25pm EST.  We have had 4 cycles of down/up connectivity changes.

DHAnderson
A model citizen

Re: Dashboard Issues?

Same here.  Email messages still pouring in.  I have clients in three different cities on several different providers.  I stopped panicking as no clients are calling me saying they are down.

 

It is interesting that the emails are in waves, switches, then access points, then cameras, finally firewalls. 

 

ZeuS41
Conversationalist

Re: Dashboard Issues?

looks like Meraki does have a lot of explaining to do, we bought into this "cloud" idea thinking the clients should not be impacted incase if the Meraki dashboard is unreachable, it turns out all VPN tunnels and clients access are linked directly to the cloud. the dashboard should only be used for management not access. if they dont fix this, we are switching our entire Meraki platform very very soon.

Wayner
Getting noticed

Re: Dashboard Issues?

I have had all of my Security appliances reporting they are going down but they are not. no other equipment has generated any emails. I have been on hold with support for 30 min so i am assuming it is a big issue for them

Twitch
Getting noticed

Re: Dashboard Issues?

Thanks, everyone. Good to know it's not just our network.

 

Other than the alerts, All is Quiet on the Western Front in terms of user complaints.

 

 

 

 

DHAnderson
A model citizen

Re: Dashboard Issues?

My clients and tunnels are still up, otherwise I would have heard. They absolutely rely on Meraki, and not one of them have reported any issues. I even called on client and they reported everything was working.  All of my clients absolutely rely on Meraki, and not one of them have reported any issues.

 

My own equipment reported that it was down, but my home office kept humming along, with emails pouring in!

 

Blueshift
Conversationalist

Re: Dashboard Issues?

Can confirm here.
Not only was cloud unreachable, logs show all of our wireless devices re-authing with the APs and all of our manually disabled ports on the switches are now re-enabled.

 

No users seem to have been affected, however.1.PNG

Aztec_Ninja
Here to help

Re: Dashboard Issues?

Did your site to site VPN go down?  

 

We have about  21 sites across the US and all our tunnels are up.  We use a separate monitoring tool to monitor.

 

It is alarming to receive all of the up/down alerts thought.  

ZeeBoussaid
Getting noticed

Re: Dashboard Issues?

same goes for us, we have 200 remote office with VPN tunnels dropped. what happened today should scare the crap out of any engineer. this is going to be a nightmare if all access to clients is tied directly to the Meraki Dashboard. i hope Meraki fix this issue or at least chime in. because im seriously considering dropping all Meraki products in our environment.
rconiv
Conversationalist

Re: Dashboard Issues?

Getting it here also, had a recorded outage from 10:52 to 11:01 PDT at both our local and colo, and then been getting some emails since for either both locations or just the colo loosing/gaining connection to the Meraki Cloud.  As others have said, nothing seems to have really died because there was no screaming.  Have been getting occasional emails since mostly for the colo.  Am on hold with Meraki, who I am pretty sure we have overwhelmed them. 

drgnslyr
Getting noticed

Re: Dashboard Issues?

Reviewing our separate network monitoring, no Site to Site tunnel issues, all stayed up, we have 50+ locations.  Haven't heard any reports of outages of issues impacting end-users, thankfully.

 

RodrigoC
Meraki Employee

Re: Dashboard Issues?

Hey Everyone,

 

Justa wanted to jump in and let everyone know that we are aware of the ongoing issues with Dashboard. As of now, we have confirmed that the issue is isolated to a small subset of servers, and as of 20ish minutes ago, service impact should be mitigated. We will share more information once we have conducted a full investigation into the issue.

 

If you are still experiencing issues, please feel free to let me know.

 

All the best,

Rodrigo

Meraki Support

Twitch
Getting noticed

Re: Dashboard Issues?

Our site-to-site VPNs have remained operational. They still show up/up and green. I haven't heard anything to the contrary in terms of remote site connectivity.

 

 

AAJ
New here

Re: Dashboard Issues?

We had the same experience. All our tunnels went down, but why ?  Meraki  always say that the network would still function without the dashboard, you just can't manage it.   Clearly that was not the case today.

 

Twitch
Getting noticed

Re: Dashboard Issues?

Hey @RodrigoC - Thanks very much for the update. We appreciate everything you guys are doing to figure-out what's going on.

DHAnderson
A model citizen

Re: Dashboard Issues?

When you say the tunnels went down, did they actually go down, and no data could be transferred, or that the dashboard reported them down?

Remo
New here

Re: Dashboard Issues?

This happened to my NOC.

Multiple sites seemed to have dropped for about 7 minutes, but there was one site that we manage that was not affected.

 

Meraki confirmed that this was only a cloud management outage as hosted services migrated to another site since the affected site was under an outage. No actual service impacted issues, just management/visibility.

I imagine you'll get the full details if you call them or open a ticket with them against one of the networks that were affected in your organization.

PDSKturley
Conversationalist

Re: Dashboard Issues?

For what it's worth, None of our Meraki VPN's or site to site VPN's actually dropped. Not a singe call from a user about disconnects or WiFi access lost.

 

While this is a serious issue, we are relieved it didn't actually affect usability. Just the management portal. Meraki will certainly have to answer the questions about why. And reassure us this platform is resilient. I would expect an explanation of the root cause, and details about the resolution for future mitigation should it happen again.

RodrigoC
Meraki Employee

Re: Dashboard Issues?

Hey @AAJ,

 

Dashboard's only role in the formation of an Auto-VPN tunnel is sharing Port and IP info with Peer MXs. A Dashboard issue like this would make previously established tunnels show offline on the portal, but remain up in actuality. Were you unable to access resources on the other side of your tunnels?

ZeeBoussaid
Getting noticed

Re: Dashboard Issues?

i can confirm what others said about the VPN drop. i was looking at the wrong logs, all VPN tunnels remained operational. but still Meraki has to mitigate the issue

RodrigoC
Meraki Employee

Re: Dashboard Issues?

Hey @Twitch,

 

Thank you for your kind words. We are deeply sorry for the inconvenience and we'll keep you all appraised as things develop. In the meantime, if anyone has any questions or concerns, feel free to DM or tag me! 

srbright
New here

Re: Dashboard Issues?

@RodrigoC What would be the recommendation on determining the real tunnel status?  Or do we have to test each tunnel comm individually on our own, which is impractical.   Are we just blind until this is resolved?  

tantony
Head in the Cloud

Re: Dashboard Issues?

Yes, I had dashboard issue for about 5 minutes, around 2 PM eastern time.

Twitch
Getting noticed

Re: Dashboard Issues?

Same here - we never lost functionality in terms of user connectivity or connectivity between sites, there was just a ton of emails saying that nothing was reachable and everything was going down all at once at all sites.

It was clearly enough to create an "Oh Sh*t" moment, and caused me to start remembering all of the changes I made earlier in the day.

 

You know the moment: "Dammit. Did I cause this?????" While sprinting away from that half-made cup of coffee back to your computer at full speed...

 

 

DHAnderson
A model citizen

Re: Dashboard Issues?

I had just cooking hot dogs for lunch when my phone lit up.  My poor hotdog boiled to death while I was pouring over email and making calls to clients.

 

 

RodrigoC
Meraki Employee

Re: Dashboard Issues?

Hey @srbright,

 

The best way to confirm if your tunnels are working would be to send some traffic across some of them. For some people, testing all tunnels might not be viable. 

 

My advice for anyone in this situation would be to test your most critical tunnels, confirm that traffic is flowing, and then send an FYI to people using those tunnels to let you know if they encounter any issues.

Twitch
Getting noticed

Re: Dashboard Issues?

@DHAnderson  Oh man, that's hilarious. RIP to your wiener. What a way to go...

 

🤣🤣🤣

RodrigoC
Meraki Employee

Re: Dashboard Issues?

Hey @DHAnderson,

 

Sorry to hear about your hot dogs 😞

 

If you are ever in Chicago (post-COVID) near the Meraki Offices, I'd be happy to treat you to a Chicago Dog!

Aztec_Ninja
Here to help

Re: Dashboard Issues?

Not the best advice during this potential outage, but I learned my lesson some years back.  We stood up PRTG to monitor our network in additional to the Merkai alerts. 

 

I still almost wet my fresh pants when seeing all of the Meraki alerts.

Twitch
Getting noticed

Re: Dashboard Issues?

@Aztec_NinjaI know that feeling! We all need some Meraki-branded Depends if this is going to keep happening.

BlakeRichardson
Kind of a big deal

Re: Dashboard Issues?

@Aztec_Ninja  +1 for PRTG, wil its expensive its very good. We use that as one of our pieces of monitoring software. 

 

There is a free version with 100 sensors. 

Meraki CMNO, Ruckus WISE, Sonicwall CSSA, Allied Telesis CASE & CAI
Bearb
Conversationalist

Re: Dashboard Issues?

Our Orgs received not only alerts for networks with VPN, but all devices in those networks with VPN produced alerts in Meraki.  We also saw Meraki alerts for our other networks without VPN or Meraki firewalls (MR's).  So this was not focused on just VPN networks.  We also monitor some of our Meraki gear with PRTG as well as snmp.meraki.com which showed down for about 8 minutes during the issues.  

 

Thankfully I'm not aware of any of our networks actually experiencing issues during this outage.  Looking forward to reading the RCA, and now Azure looks to want to join in on the Monday fun with their own issues.  Happy Monday everyone!

rconiv
Conversationalist

Re: Dashboard Issues?

Just saw that Twitter alert a bit ago about M365 issues.  I can't get in to the admin portal right now, either give a not available, or only loads a bit of it.

 

(Edit) Though now it works.  That was pretty quick for them.

DHAnderson
A model citizen

Re: Dashboard Issues?

O365 was down due to Azure Active Directory having issues.  That also brought Cisco Duo to send me this email subject line:

 

Azure Conditional Access Authentication failures.

 

It is truly the ides of March!

 

nb12345
Conversationalist

Re: Dashboard Issues?

Hello,

 

Did you confirm the RCA? what was the reason for outage please?

 

Thanks,

lpopejoy
Building a reputation

Re: Dashboard Issues?

"Meraki-branded Depends"  

 

...You know, they are big on swag...  😂

Twitch
Getting noticed

Re: Dashboard Issues?

@lpopejoy   😂🤣 

 

Could you imagine? Rock Your Depends Day at the office. Heaven help us...

 

 

RodrigoC
Meraki Employee

Re: Dashboard Issues?

Hey Everyone!

 

Checking in to provide a quick update. We are still working on putting together a full public RCA  to share with you all. Due to the nature of the issue, this might take several more days.

 

That said, I wanted to share what we know for certain at this time.

 

There was an outage on March 15th that affected customers hosted in the Americas region. The outage started 10:55 PT and lasted 8 minutes. During this time, some customers were not able to log into or access Dashboard. All services were restored at 11:03 PT.

The outage was due to a core network device managed by one of our data center service providers that needed to be rebooted and reconfigured. We are working with the provider to ensure this type of issue does not happen in the future.

Dashboard (dashboard.meraki.com) was unavailable during this time, however, there was no disruption to network connectivity and services at customer locations. Meraki devices were online throughout, though they will report as being offline during that time.

 

I hope this proves helpful and as always, please don't hesitate to reach out if there is anything else I can clarify for you.

 

Happy Tuesday to you all! 😃 

View solution in original post

ITDept-GT
New here

Re: Dashboard Issues?

Is there a status page or email list that we can subscribe to, to be notified of these events?

Welcome to the Meraki Community!
To start contributing, simply sign in with your Cisco account. If you don't yet have a Cisco account, you can sign up.