//Dashboard trouble today - 8/30/2020

[RU-JK]

Can anyone else confirm there is a dashboard issue today?  Got all the all the alerts, but I can access equipment remotely.  I am not bale to login to dashboard.  I am currently on hold with support to confirm as well.  it is multiple sites i was alerted on at same time.

[Schmitty]

Yes, numerous customers are here posting the same issue(including myself).

I also, validated all our devices are up and functioning its just the loss to the dashboard from what I am seeing.

[RU-JK]

Thank you for the reply..... in some respects, glad its not just me, but also yikes, its not just me....  wish there was a true status page...  not how wanted to spend my Sunday morning...

[RU-JK]

I am starting to see services do the up/down alerting now.... hopefully we are on the path to a resolution (no pun intended).

On a side note, I cannot confirm or deny this but I am hearing that this was due to the United States upgrading the Internet to handle all the Labor Day Mattress Sales next weekend.  😃

[RU-JK]

Just got my confirmation form Meraki Support, they will add any new cases to their internal case and it should update users when they post updates...  Suggest you send to support@meraki.com and then you can get auto-notifications....

[Pickpacket]

Yes, all kinds of alerts this morning about dashboard and APs going down.  Dashboard is intermittently accessible, trying to make some changes this morning and virtually impossible at the moment.

[RU-JK]

Thank you... yes,  kind a freaked me out...  all my sites at once, figured it was a dashboard issue not my issue...  wanted to confirm I was not a one off issue

[GARA1]

I have the same issue.

When I call the 1 (415) 937-6671 number I get what sounds like lots of packet loss.

Is Meraki under attack from hackers?

[Schmitty]

No this seems to be a massive ISP issue.

[GARA1]

I figured it was a large scale internet issue as I am having tunnel failure issues also, that is the tunnels provisioned by Meraki.

[RU-JK]

overall I am good with Meraki... Just think they need a better way to communicate service disruptions.  Certainly if a bunch of end users can talk about it then they should be able to have a posting somewhere two hours after it started...

[CurtisN]

Agree, this makes it a touch difficult to finish installing new AP's today.

[Dasilva13]

https://twitter.com/HIVELOCITY?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor

\

appears like centurylink (level 3) worldwide issue

[RU-JK]

yes, i am starting to see other companies now posting... i suppose better to happen on a sunday than monday , but my org is healthcare and 24/7  .. now to field all the tix of  site xyz.com is not working....

[PhilE]

Hello everyone,

Thank you for bringing this up.

I can confirm this is part of a wider internet outage and isn't Meraki specific.

You can see additional information on the outage here and here

Edit: Just to confirm, I do work for Meraki but it will take a few days for my account to be verified.

[PattyB]

Is there a Meraki status page that to view updates on the outage?  Thanks 

[PhilE]

Hi PattyB,

There isn't a status page at this time, however, the outage is not Meraki specific.

I suggest keeping an eye on the CenturyLink status page for updates

[PattyB]

Thanks.  Does Meraki have a backup to CenturyLInk? As customer, I have more than one internet provider to fail over to in the event one goes down. 

[PhilE]

Hi PattyB,

We do have redundancy but due to the nature of the issue, multiple carriers are affected.

Thanks

[RU-JK]

I gave up being on hold with support, I did email just to have it logged.  My dashboard comes and goes at the moment.  For other services I have alerts today from Lastpass, Microsoft, and Xerox....  DNS, its always DNS... 😃

[WPOOLEIEC]

They said it was resolved, but I still can't get on. 

[OnwardSolutions]

The more I work with Cisco Meraki the more I realize this company is complete garbage. Expensive garbage.

[Schmitty]

@OnwardSolutions, super harsh considering this is a wide spread internet service issue. 

Granted if the community site can be up why not the actual meat and potatoes piece of the solution be up and not impacted. 

I get frustrated with Meraki plenty. I feel calling the support desk most of the time gets me nowhere fast if its something even a little bit in the grey(outside the Meraki cookie cutter shape).

I'm no White Knight but, I am pragmatic. There are numerous internet based services affected right now. 

I think that is one of the gotcha items of the oh so magical cloud. If there is an issue on the internet it's going to affect the solution. 

[OnwardSolutions]

As much as we pay for this crap it's completely merited please spare me this wide spread internet service garbage. This is the type of behavior that enables company like Cisco to drag their feet on accessibility features. This is not acceptable. No status pages to advise what's going on, no ability to call into support without access to the dashboard or memorizing your customer number.. Cisco Meraki is a dying breed. Cost effective hybrid solutions are taking over this space. What's the advantage of cloud-based networking when the cloud is just another single point of failure?

[Schmitty]

@OnwardSolutions sounds like you have some project planning and budget dollars to secure friend.

Good luck.

P.S.

You do know internet outages are real though right?

Much like a couple years ago a Comcast BGP change brought down the entire east coast of the US internet services. Now image one company who has the backbone routing in many different parts of the country and world having this issue. 

[OnwardSolutions]

@Schmitty  We most certainly do.

Internet outages we can be dealt with, we have fail-over circuits for that. Loosing the ability to manage your infrastructure on top of a wide spread internet outage is an embarrassment. 

I'll repeat, For as much as this garbage costs, lacking the ability to manage your infrastructure because the "cloud" is down is NOT acceptable. It's 2020. Cisco Meraki is a laggard in this space. Both my organization and client's are paying the price due to the inadequate releases of features and quality of life improvements from Cisco Meraki.

Your scapegoat argument just doesn't hold any weight bud, sorry. 

[Schmitty]

@OnwardSolutions as I said, good luck. 

Not sure what 2020 has to do with it. BGP and router interconnectivity has not changed. 

Instead of being angry here where this is a community of end users assisting others like information sharing on this outage. 

Shoot your Cisco rep and email and let them know. 

Good luck again. Conversions can always be a headache. 

[OnwardSolutions]

@Schmitty  I will find any avenue I can to voice my opinions as it's the only way it's going to drive any real change.

Your failure to acknowledge the premise of my argument is the problem. You keep referencing internet outages and BGP as if it has anything to do with Cisco Meraki's inability to provide it's customers with out-of-band management. Stop fanboying this subject into oblivion and let's call it for what it is, garbage. Cisco Meraki has fallen behind the curve and it continues to do so.

[GIdenJoe]

There most certainly were alot if internet issues today, even for residential customers.

My kids could find online games for nintendo switch games but had a 10% chance of succesfully connecting to a game,  So should I now announce to the world that the nintendo switch online is crappy because there are internet routing issues?  I had trouble reaching a multitude of sites too.

There is cloud redundancy per region but if routing inside the tier 1 providers just prohibit your access to the dashboard that is outside of Meraki control and you should not blame them for an issue of this magnitude.

I'm very critical towards Meraki on alot of things but your rage thinking is uncalled for.

[Schmitty]

@OnwardSolutions please do not put words in my mouth or out of my finger tips. I am not even close to being a fanboy of Meraki. You must have missed my previous post calling their support next to useless. 

Nor did I ever say I agreed with the fact the primary interface for the solution is down while the community page is up. 

Again, I am simply a pragmatic person. Who also had a change that I had to post pone this morning. My point is simple mulling and complaining here is equal to screaming into the wind. 

I did express my more negative and disappointed opinions to our Cisco rep via email asking for this to be submitted for further review. 

Feel free to continue on and rage against the machine in a wind tunnel. 

Truly good luck to you and anyone else in your project in replacing this solution. 

[EdwardDybas]

What suggestions do you have instead of using the Meraki cloud?  Our organization too was at the mercy of the Meraki non-redundant cloud to be used for a mission critical medical field, and our home based employees could not connect in to VPN this morning.

Knowing now that Cisco Meraki is unable to meet our needs of a suitable disaster recovery, I see no choice but to move to another platform, despite CenturyLink's woes.

One thing is certain, I do not trust the availability of the Meraki cloud going forward, and that's quite a blow to their reputation.

[Schmitty]

So devices themselves would still function as configured during this as long as they had connectivity to the internet to create their connection. 

What was down this morning was the management plane. Not the data plane unless you had changes to make. 

Again I do not agree that this is acceptable but, it is a hazard of the "cloud". Cloud = your stuff in someone else's data center. 

[EdwardDybas]

It wasn't just management that was down.  It was  client VPN.  That's my issue.

[ww]

@EdwardDybas you are using meraki auth  and/or the dns for clientvpn?

[EdwardDybas]

@ww yes to both. 

[PhilE · Accepted solution]

Hi Everyone,

Thank you for your feedback. Firstly, as mentioned above, I would encourage you to feed this back to your sales representatives as another avenue.

Secondly, the above points have been flagged for review and we will have a more detailed response to address your concerns in the post mortem which I will share here as well.

Thanks,

Phil

[OnwardSolutions]

With all due respect @PhilE, this feedback has been provided to our multiple sales reps and the "request a feature" links on the dashboard for as long as we have resold Meraki (10 years+).

Development and feature adoption is crawling at a snails pace. We are being left behind by competitive more cost effective solutions. 

Please forgive me for being pessimistic but it seems as if Meraki is more invested in marketing and pushing out "SWAG" more than DevOps and driving for industry leading network tech. 

In any event, I hope this latest event is the catalyst that pushes for real change.

[EdwardDybas]

I would be happy if when using "Meraki Cloud" authentication for VPN, it's possible to modify the process to use a cached copy of the credentials on the MX appliance directly, any time it cannot reach the cloud.

It was a major inconvenience to lose VPN connectivity on Sunday.

[DavidF1]

I'd be happy to be able to manage our devices "out of band" when cloud control fails.  My guess is this will never be allowed to happen, and thus now planning our exit from Meraki.  

[EdwardDybas]

Our Meraki Cloud's VPN authentication is down again.  Please fix this ASAP!

[DavidF1]

Agree 100%.  Seeing people deflect this as an Internet issue and not Meraki is BS.  Just like when Meraki deleted their entire object store and had no backups for it.

This is the final straw for us.  We are a healthcare customer and can't operate like this, especially during a crisis that has everyone in our company on edge.