Hosted Services Issues

[GiacomoS]

Good day everyone, 

We are currently experiencing an issue with multiple hosted services and configuration fetches from Dashboard. 

The impacted services would include:

- Device configuration fetch (all devices)

- Client VPN with Meraki Authentication

- Systems Manager enrollment (if gated behind Meraki Authentication)

- SSID scheduling

- Any other service using Meraki Authentication

- Site to Site VPNs in a few shards

Our teams are aware and are working to resolve this as promptly as possible. 

The waiting times calling in are currently extended. 

We will provide further updates here as we receive them. 

Many thanks.

Giac

Cisco Meraki Support

[Jayt]

Once this gets sorted out, I'd like to know why this is effecting our local SSID clients from connecting via our local NAT network.  New to Meraki.

Thanks

[sixgigdan]

Do we have any kind of ETA on successful resolution of this issue?

Also.....WHY DON'T YOU HAVE A STATUS PAGE?!

[RockHopper]

Agreed. Such a simple thing as most all cloud providers have. 

[GiacomoS]

Hey @sixgigdan , 

Not yet unfortunately, but we have multiple teams assisting to resolve this as soon as possible. 

A root cause has been identified and we are currently working on a fix.

I hear your feedback with the Status page and will ensure I raise it up.

Many thanks for your patience everyone.

Giac

Meraki Support

[TrevorB]

Not having an ETA for essentially a full stop of service is incredibly concerning. Some businesses have no other recourse in accessing local resources in areas with a WFH order. There are no status updates, and nothing to indicate progress aside from a green banner at the top of the cloud portal. I have nothing to provide our team in regards to when to expect service restoration.

[micronet_info]

We have a venue with more than 1000 guest with no internet connection and can't take any payments as we don't have any internet.

2x Meraki MX 250 and 8 x Meraki MS250-48LP along with 20+ AP's are all useless as firewalls are out of config and we can't bring them online (firewall rules). 

@GiacomoS We need an ETA and get this fixed as soon as possible.

[Inderdeep]

@micronet_info : Meraki is all together working on the issue on high priority and will soon update with the ETA @GiacomoS  

[abdunin]

And why is the alert for this outage on the Meraki dashboard in green? That typically indicates a non-critical message. We're currently unable to configure our firewalls, config out of date for the last 45 mins.

[itxpt]

This is taking way too long, for a top tier support company this should have been fixed by now, lots of folks riding on this and are now dead in the water.

[GiacomoS]

Hey @itxpt ,

We hear you loud and clear. We have dedicated all the resources to get this resolved and it is treated with the outmost urgency. 

I look forward to get you all some good news as soon as possible.

Giac

Meraki Support

[micronet_info]

Perfect timing where our firewalls were pushing a change and now dashboard not available.

Our business is greatly affected by this as firewalls can't replicate with dashboard, we don't have any internet.

We need an ETA and a status page ASAP.

[Techbull]

@GiacomoS 

I assume you may create a list of issues related to this outage so I would like to add that all my SSID's that are tied to a schedule have not been enabled during their scheduled time. However, all the ones that didn't have a schedule were unaffected. 

I'd also like to add a vote to the status page, it would be much more helpful than the community for this type of issue.

[Inderdeep]

@Techbull : I would recommend, Lets list out all issues from everybody here and will check is that related to the hosted services issue. 

[GiacomoS]

Hey @Techbull , 

The SSID availability is pushed from Cloud , so it is going to be impacted by the outage at this moment.

Thank you for adding it here!

Giac

Meraki Support

[JRMcGee]

@itxpt 

I agree!! Our wifi is still down......

[GiacomoS]

Hello @abdunin ,

I hear you and passed the feedback to our Dashboard team. 

Thank you for your patience.

Giac

Meraki Support

[AlphacomItalia]

How long do you think it will be solved?

[itxpt]

From Meraki Support:

Hello Steve

I am really sorry to hear that your attorney's are impacted by this issue, we don't currently have an ETA. I would strongly encourage you to seek an alternative form of having colleagues meet if possible, I am sorry for the inconvenience this has caused you and your team.

Thank you, and I look forward to your response!

...go to the cloud they said, it'll be great...ha!

[abdunin]

This has to be related, a customers MX250 just completely went unreacheable, we could not reach it from the dashboard or ping it by the local LAN IP. We had to reboot it for the internet to come back on at the location and for it to be reachable via LAN. 

Now the light is rainbowing but internet is working. I'm assuming its trying to get the config.

[Kluck]

Would this issue cause client VPN authentication to fail?

[itxpt]

Client VPN is affected.

[blrh]

Since this issue is not resolved, can we remove this from being "Solved"? This is the only public source of information without having to wait an hour+ to get an answer from Meraki Support. 

[GiacomoS]

Thank you for pointing it out @blrh , rectified.

[Jayt]

I'm new to Meraki. Only use AP's but we use 800 of them in our county .gov network.  I'm not impressed with communication so far.

[Inderdeep]

@Jayt : I understand your pain, the issue is its world wide and so many customers are effective, so taking time, Be patience, soon somebody come up to you !

[Jayt]

This is exactly my problem. The SSID's with no schedule are working fine.

"all my SSID's that are tied to a schedule have not been enabled during their scheduled time. However, all the ones that didn't have a schedule were unaffected. "

[Dromero]

all my VPN: tunnel data to a concentrator ssid continue failing

[Maurice_Fellay]

Hi, does this issue also affect Site 2 Site VPN with Meraki appliances on different organizations and Site 2 Site VPN to non meraki appiances?

[JR_NYC]

We have a site to site between two MX84's and the link has been fine.

[Ayala]

Hello, does it affect new devices communicating to the Meraki Cloud Services.  I received my replacement MR33 and its not communicating to the Cloud.  Thank you, much appreciated. 

[Inderdeep]

@Ayala : The issue on the below services as defined earlier as well 

- Device configuration fetch (all devices)

- Client VPN with Meraki Authentication

- Systems Manager enrollment (if gated behind Meraki Authentication)

- SSID scheduling

- Any other service using Meraki Authentication

rest if you have other issue, it may or may not be due to Hosted services

[GiacomoS]

@Maurice_Fellay @Dromero , VPNs on some nodes have also been impacted. Tunnelled SSIDs and Non-Meraki tunnels would be subjected to the outage.

Existing tunnels should continue to operate normally. If they are not, I'd recommend opening a case so we can verify. 

@Ayala , yes, most likely. The MR will not be able to receive the configuration from the Cloud and is therefore unlikely to come online on Dashboard. 

[Maurice_Fellay]

Since the TTL time on site 2 site tunnels on default is 8 hours (28800 secs) will the simple fail if the outages continues over those 8 hours right, because some VPNs are dying while others still work. 

Since when do we have these problems? exactly???? so I can prepare our clients.

Thanks

[collabie14]

That's an excellent question.  I just tried to spin up a new hub and the IP isn't being pushed to spokes because of outage.  Fingers crossed the cloud isn't required for the tunnels to re-negotiate after the TTL expires or a circuit outage.  If not, this is gonna get real interesting real quick. 

[JulienV]

It has been 4 hours now. This was a bad timing for NA East. Can we get an ETA at least?

[amacaya]

We have dropped the VPN for 26 hours.

[TrevorB]

Half a day of billables has been lost for EST. The fact that even a rough window can't be provided so we can inform our clients is ridiculous.

[RazzleDazzle]

Agreed. Also in NA EST. Meraki needs to have a solid ETA and needs to provide a status page moving forward. This is a very poor reflection on Meraki's reliability and service. Not the service expected for the prices Meraki charges. 

[GiacomoS]

@Maurice_Fellay , it would very much be subject to when they rekeyed last.

The issues has surfaced around 2:00 PM BST

Many thanks. 

Giac

Meraki Support

[micronet_info]

@GiacomoS  any updates regarding the resolution or ETA for fix to be implemented?

[amacaya]

- Client VPN with Meraki Authentication, We declared the problem 26 hours ago, case 06834937. A faster solution is required. other organizations begin to be affected.

[GiacomoS]

@amacaya , your original case may not be related to the outage here, but it will likely be impacted throughout today.

Please continue to work with our Support team to continue investigating what the root cause would be. 

Many thanks.

Giac

Meraki Support

[wrespawn]

Throwing in that my SSID is down as well. Last year I was almost sold on switching all our P2P VPN devices with Meraki ones. Decided to renew with what we had, I am so glad I dont have 27 stores offline right now. Just a little bit of WIFI outage isnt brutal for this organization. I will remember this day for years to come next time I get a sales call from Meraki. For sure impacted future business. 

[BigMtnIT]

Starting to get a flood of calls from my customers for Client VPN authentication errors.  Site to Site still working thankfully.  We need this resolved ASAP.  .

[GiacomoS]

Hi all,

We'll start marking updates as "Solutions" so we can have them at the top of the thread. 

Please be on the lookout for them.

Many thanks.

Giac

Meraki Support

[GrahamG]

This is less than reassuring. The first image shows the current dashboard alert message, the second shows the alert message from a couple of hours ago. What happened to "Meraki devices and networks will continue functioning normally "?

Current alert messageAlert message from 8 am PST

[krolunger]

My email and my phone won't stop ringing with questions and anger. Please resolve this promptly. 

[MDR17]

All Meraki customers' Overview map should be showing applicable outage indications (Unavailable SSIDs, Failed Policies, etc.)...  Our Overview map shows ALL GREEN which is NOT correct!  Meraki needs to update the Overview map to reflect correct system statuses for these types of issues.

[JulienV]

VPN looks back for my clients

[GrahamG]

Same here. VPN just connected

[GiacomoS]

Good day everyone,

Our development teams have successfully identified the root cause of the issue and are continuing to work on implementing a fix. 

We understand your frustration and appreciate your continued patience and support during this time. 

[gtatech]

I was able to connect finally and can see now that life may return back to normal.  I hope this never happens again!

[RajuD]

We are able to connect as well.

We have so many questions:

1. Why was there no status page ?

2. Why was no one from Meraki responding on social media about the outage?

3. It took 1 hr 40 mins to get Meraki support on phone just to confirm the issue. It could have been proactively.

4. Why was the change pushed during business hours (if this was caused by change), ? I dont seem to recall any notice about scheduled upgrades.

[srttallhat]

Though I appreciate the PR level of vague updating, We will need an adequate evaluation on what broke, why it broke, and how it can be prevented in the future.  It may be steep, but this seemed to have been a meraki wide outage, and due to the nature of more and more companies relying on remoting services, this 5 hour outage has costed millions in disturbances.

[JRMcGee]

@GiacomoS 

As long time Meraki customer, why did it take 5+ hours to fix this issue. What was the root cause? How is Meraki going to prevent this from happening again? Is there a plan in place to remedy issues like this? This is very concerning that this has happened. 

[Maurice_Fellay]

Site to Site - Non Meraki VPNs are working again.

[MK2]

I'm really surprised, almost every cloud product from cisco now has a status page where even an incident can be tracked transparently for everyone.

Examples?:
https://status.webex.com/service/status?lang=en_US
https://status.umbrella.com/#/

Why don't you have something like that???

[GiacomoS]

@MK2 we hear you and we are passing the feedback on internally. 

Thank you for sharing it!

Giac

[GiacomoS]

Community,

We'd like to thank you all for being so understanding and supportive. We completely understand how difficult the situation has been for many of you. We want you to have a great experience when you use our platform and our products and we are committed to ensure that you do. 

Please be on the lookout for comms via the cases and engage with our Support teams when necessary. 

Keep the feedback coming, it never falls on deaf ears.

Thank you again for all your patience and engagement!

Giac

[NordOps]

One other suggestion. 

This isn't the first time there has been a backend cloud issue and when you call in and wait on hold forever it doesn't help anyone's patience.  It might go a long way to announce an issue like this before putting the call into the support que.  Being able to call Meraki support easily is one of the best features, but being on hold for 45 minutes waiting to talk to someone while back end issues are being worked on by engineers is extremely frustrating.

[SteveB2021]

I second that.

Had the same experience too.

The hold music should be changed to an on hold message stating theirs and outage.

[GiacomoS]

@SteveB2021 @NordOps thank you for sharing!

You are spot on there and I'm happy to share that this is something that we have included in our process. Unfortunately due to some technical difficulties, the announcement was injected after a few hours from the start of the outage and was not immediately played for those who were already in the queue waiting to speak with us. 

Many thanks!

Giac

[r393292492]

Thank you Meraki, my life is now ruined. My wife is walking out the door with the kids, my dog ran away, and my family has disowned me. All I can hear is the constant ringing of my phone and banging on the door of the server closet. I'm scared for my life. Will anyone save me? What will you do to fix this? Will it ever end?

[RazzleDazzle]

Don't forget the angry mob with torches and pitchforks outside your door that is setting fire your home and has brought a vat of boiling Meraki™ oil, just for you. 

[Maurice_Fellay]

Life really sucks when your dog ran away, all other issues you mention are just the most common situation for an IT guy.

But hey, you can always adopt a new dog.

[Maurice_Fellay]

Client VPNs are starting to authenticate, Still changes to dashboard are not being sent to appliances. but at least we are starting to get into normal operation.

Will Meraki send information on what was the issue, why did it happen and what will be the remediation in place so it will not happen again?

[GiacomoS]

@Maurice_Fellay  @srttallhat , please make sure you raise a Dashboard case to request a reason for outage.

Many thanks.

Giac
Meraki Support