- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Dashboard Issues?
Anyone else seeing weird issues with the Dashboard today? I was able to access it just fine earlier today, but then access to the it just went belly-up. Not long after that, I began receiving all of these notifications that devices are going down at all sites, but when I look at them in the Dashboard (which became reachable again) they appear to be just fine. The fact that my email and phone are not blowing-up with people in a panic seems to confirm this.
Are there any known-issues at the moment??
Thanks!
Twitch
Solved! Go to solution.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hey Everyone!
Checking in to provide a quick update. We are still working on putting together a full public RCA to share with you all. Due to the nature of the issue, this might take several more days.
That said, I wanted to share what we know for certain at this time.
There was an outage on March 15th that affected customers hosted in the Americas region. The outage started 10:55 PT and lasted 8 minutes. During this time, some customers were not able to log into or access Dashboard. All services were restored at 11:03 PT.
The outage was due to a core network device managed by one of our data center service providers that needed to be rebooted and reconfigured. We are working with the provider to ensure this type of issue does not happen in the future.
Dashboard (dashboard.meraki.com) was unavailable during this time, however, there was no disruption to network connectivity and services at customer locations. Meraki devices were online throughout, though they will report as being offline during that time.
I hope this proves helpful and as always, please don't hesitate to reach out if there is anything else I can clarify for you.
Happy Tuesday to you all! 😃
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have gotten >70 alerts in 10 minutes for three organizations across three states and close to 40 networks. All other monitoring I have in place show nothing but alerts from Meraki are going nuts.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Same issue here. Waiting to get TAC on the phone so they can confirm. Last time we experienced this, the root cause was maintenance being done on the Meraki Network my company resides in.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yeah it appears that the Dashboard was not accessible for 10ish minutes. In that time frame devices were not able to check in which is triggering the email messages about devices being offline when in fact they have been working fine the whole time.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yep, same thing here!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I got kicked out of the dashboard and then couldnt get logged back in. Now I am getting a ton of email alerts about my VPN tunnels being down.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We had the same experience. All our tunnels went down, but why ? Meraki always say that the network would still function without the dashboard, you just can't manage it. Clearly that was not the case today.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
When you say the tunnels went down, did they actually go down, and no data could be transferred, or that the dashboard reported them down?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hey @AAJ,
Dashboard's only role in the formation of an Auto-VPN tunnel is sharing Port and IP info with Peer MXs. A Dashboard issue like this would make previously established tunnels show offline on the portal, but remain up in actuality. Were you unable to access resources on the other side of your tunnels?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@RodrigoC What would be the recommendation on determining the real tunnel status? Or do we have to test each tunnel comm individually on our own, which is impractical. Are we just blind until this is resolved?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hey @srbright,
The best way to confirm if your tunnels are working would be to send some traffic across some of them. For some people, testing all tunnels might not be viable.
My advice for anyone in this situation would be to test your most critical tunnels, confirm that traffic is flowing, and then send an FYI to people using those tunnels to let you know if they encounter any issues.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
Did you confirm the RCA? what was the reason for outage please?
Thanks,
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Not the best advice during this potential outage, but I learned my lesson some years back. We stood up PRTG to monitor our network in additional to the Merkai alerts.
I still almost wet my fresh pants when seeing all of the Meraki alerts.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Aztec_NinjaI know that feeling! We all need some Meraki-branded Depends if this is going to keep happening.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
"Meraki-branded Depends"
...You know, they are big on swag... 😂
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hey Everyone!
Checking in to provide a quick update. We are still working on putting together a full public RCA to share with you all. Due to the nature of the issue, this might take several more days.
That said, I wanted to share what we know for certain at this time.
There was an outage on March 15th that affected customers hosted in the Americas region. The outage started 10:55 PT and lasted 8 minutes. During this time, some customers were not able to log into or access Dashboard. All services were restored at 11:03 PT.
The outage was due to a core network device managed by one of our data center service providers that needed to be rebooted and reconfigured. We are working with the provider to ensure this type of issue does not happen in the future.
Dashboard (dashboard.meraki.com) was unavailable during this time, however, there was no disruption to network connectivity and services at customer locations. Meraki devices were online throughout, though they will report as being offline during that time.
I hope this proves helpful and as always, please don't hesitate to reach out if there is anything else I can clarify for you.
Happy Tuesday to you all! 😃
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Is there a status page or email list that we can subscribe to, to be notified of these events?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The Dashboard seems to be down again.
I am able to sign in, select a customer, but the page hangs or takes an extremely long time to display.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
A bit ago when I opened the dashboard, and it logged me in, it hung just showing the Meraki logo, and Health. Nothing else was shown, though if I clicked on one of the news articles, it would show the full page. After a bit though the issue went away. I would select a users VPN and it works fine, though that might not be the same thing as what you are selecting. Outside of the one page, everything else came up at their normal speed.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
It has come back up for me now. It must be transient issues and it has happened a couple of times now.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Seems to be working for me on the mobile meraki app?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Aztec_Ninja +1 for PRTG, wil its expensive its very good. We use that as one of our pieces of monitoring software.
There is a free version with 100 sensors.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Experienced the same thing, from 10:54 AM to 11AM.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
yeah, my NOC is pissed, 200+ tickets in 20 minutes, and no legit known issues. Not sure what broke on Meraki's backend, but that is really annoying on a Monday.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Same for us, all our VPN tunnels dropped.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Same here, getting all sorts of Network down alerts. I just got a 2nd wave of alerts. The dashboard looks good, though.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Same here. Email messages still pouring in. I have clients in three different cities on several different providers. I stopped panicking as no clients are calling me saying they are down.
It is interesting that the emails are in waves, switches, then access points, then cameras, finally firewalls.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Same here!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We are having the same issues. All of our Meraki networks' devices are losing connectivity (down/up) with the Meraki Cloud. This started for us at 1:50pm EST and the last loss of connectivity was at 2:25pm EST. We have had 4 cycles of down/up connectivity changes.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
looks like Meraki does have a lot of explaining to do, we bought into this "cloud" idea thinking the clients should not be impacted incase if the Meraki dashboard is unreachable, it turns out all VPN tunnels and clients access are linked directly to the cloud. the dashboard should only be used for management not access. if they dont fix this, we are switching our entire Meraki platform very very soon.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
My clients and tunnels are still up, otherwise I would have heard. They absolutely rely on Meraki, and not one of them have reported any issues. I even called on client and they reported everything was working. All of my clients absolutely rely on Meraki, and not one of them have reported any issues.
My own equipment reported that it was down, but my home office kept humming along, with emails pouring in!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Did your site to site VPN go down?
We have about 21 sites across the US and all our tunnels are up. We use a separate monitoring tool to monitor.
It is alarming to receive all of the up/down alerts thought.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Reviewing our separate network monitoring, no Site to Site tunnel issues, all stayed up, we have 50+ locations. Haven't heard any reports of outages of issues impacting end-users, thankfully.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hey Everyone,
Justa wanted to jump in and let everyone know that we are aware of the ongoing issues with Dashboard. As of now, we have confirmed that the issue is isolated to a small subset of servers, and as of 20ish minutes ago, service impact should be mitigated. We will share more information once we have conducted a full investigation into the issue.
If you are still experiencing issues, please feel free to let me know.
All the best,
Rodrigo
Meraki Support
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hey @RodrigoC - Thanks very much for the update. We appreciate everything you guys are doing to figure-out what's going on.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hey @Twitch,
Thank you for your kind words. We are deeply sorry for the inconvenience and we'll keep you all appraised as things develop. In the meantime, if anyone has any questions or concerns, feel free to DM or tag me!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Our site-to-site VPNs have remained operational. They still show up/up and green. I haven't heard anything to the contrary in terms of remote site connectivity.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have had all of my Security appliances reporting they are going down but they are not. no other equipment has generated any emails. I have been on hold with support for 30 min so i am assuming it is a big issue for them
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks, everyone. Good to know it's not just our network.
Other than the alerts, All is Quiet on the Western Front in terms of user complaints.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can confirm here.
Not only was cloud unreachable, logs show all of our wireless devices re-authing with the APs and all of our manually disabled ports on the switches are now re-enabled.
No users seem to have been affected, however.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Getting it here also, had a recorded outage from 10:52 to 11:01 PDT at both our local and colo, and then been getting some emails since for either both locations or just the colo loosing/gaining connection to the Meraki Cloud. As others have said, nothing seems to have really died because there was no screaming. Have been getting occasional emails since mostly for the colo. Am on hold with Meraki, who I am pretty sure we have overwhelmed them.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This happened to my NOC.
Multiple sites seemed to have dropped for about 7 minutes, but there was one site that we manage that was not affected.
Meraki confirmed that this was only a cloud management outage as hosted services migrated to another site since the affected site was under an outage. No actual service impacted issues, just management/visibility.
I imagine you'll get the full details if you call them or open a ticket with them against one of the networks that were affected in your organization.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
For what it's worth, None of our Meraki VPN's or site to site VPN's actually dropped. Not a singe call from a user about disconnects or WiFi access lost.
While this is a serious issue, we are relieved it didn't actually affect usability. Just the management portal. Meraki will certainly have to answer the questions about why. And reassure us this platform is resilient. I would expect an explanation of the root cause, and details about the resolution for future mitigation should it happen again.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Same here - we never lost functionality in terms of user connectivity or connectivity between sites, there was just a ton of emails saying that nothing was reachable and everything was going down all at once at all sites.
It was clearly enough to create an "Oh Sh*t" moment, and caused me to start remembering all of the changes I made earlier in the day.
You know the moment: "Dammit. Did I cause this?????" While sprinting away from that half-made cup of coffee back to your computer at full speed...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I had just cooking hot dogs for lunch when my phone lit up. My poor hotdog boiled to death while I was pouring over email and making calls to clients.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hey @DHAnderson,
Sorry to hear about your hot dogs 😞
If you are ever in Chicago (post-COVID) near the Meraki Offices, I'd be happy to treat you to a Chicago Dog!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
i can confirm what others said about the VPN drop. i was looking at the wrong logs, all VPN tunnels remained operational. but still Meraki has to mitigate the issue
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes, I had dashboard issue for about 5 minutes, around 2 PM eastern time.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Our Orgs received not only alerts for networks with VPN, but all devices in those networks with VPN produced alerts in Meraki. We also saw Meraki alerts for our other networks without VPN or Meraki firewalls (MR's). So this was not focused on just VPN networks. We also monitor some of our Meraki gear with PRTG as well as snmp.meraki.com which showed down for about 8 minutes during the issues.
Thankfully I'm not aware of any of our networks actually experiencing issues during this outage. Looking forward to reading the RCA, and now Azure looks to want to join in on the Monday fun with their own issues. Happy Monday everyone!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Just saw that Twitter alert a bit ago about M365 issues. I can't get in to the admin portal right now, either give a not available, or only loads a bit of it.
(Edit) Though now it works. That was pretty quick for them.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
O365 was down due to Azure Active Directory having issues. That also brought Cisco Duo to send me this email subject line:
Azure Conditional Access Authentication failures.
It is truly the ides of March!