Hosted Services Issues

[GiacomoS]

Good day everyone, 

We are currently experiencing an issue with multiple hosted services and configuration fetches from Dashboard. 

The impacted services would include:

- Device configuration fetch (all devices)

- Client VPN with Meraki Authentication

- Systems Manager enrollment (if gated behind Meraki Authentication)

- SSID scheduling

- Any other service using Meraki Authentication

- Site to Site VPNs in a few shards

Our teams are aware and are working to resolve this as promptly as possible. 

The waiting times calling in are currently extended. 

We will provide further updates here as we receive them. 

Many thanks.

Giac

Cisco Meraki Support

[Jayt]

Once this gets sorted out, I'd like to know why this is effecting our local SSID clients from connecting via our local NAT network.  New to Meraki.

Thanks

[sixgigdan]

Do we have any kind of ETA on successful resolution of this issue?

Also.....WHY DON'T YOU HAVE A STATUS PAGE?!

[RockHopper]

Agreed. Such a simple thing as most all cloud providers have. 

[GiacomoS]

Hey @sixgigdan , 

Not yet unfortunately, but we have multiple teams assisting to resolve this as soon as possible. 

A root cause has been identified and we are currently working on a fix.

I hear your feedback with the Status page and will ensure I raise it up.

Many thanks for your patience everyone.

Giac

Meraki Support

[TrevorB]

Not having an ETA for essentially a full stop of service is incredibly concerning. Some businesses have no other recourse in accessing local resources in areas with a WFH order. There are no status updates, and nothing to indicate progress aside from a green banner at the top of the cloud portal. I have nothing to provide our team in regards to when to expect service restoration.

[micronet_info]

We have a venue with more than 1000 guest with no internet connection and can't take any payments as we don't have any internet.

2x Meraki MX 250 and 8 x Meraki MS250-48LP along with 20+ AP's are all useless as firewalls are out of config and we can't bring them online (firewall rules). 

@GiacomoS We need an ETA and get this fixed as soon as possible.

[Inderdeep]

@micronet_info : Meraki is all together working on the issue on high priority and will soon update with the ETA @GiacomoS  

[abdunin]

And why is the alert for this outage on the Meraki dashboard in green? That typically indicates a non-critical message. We're currently unable to configure our firewalls, config out of date for the last 45 mins.

[itxpt]

This is taking way too long, for a top tier support company this should have been fixed by now, lots of folks riding on this and are now dead in the water.

[GiacomoS]

Hey @itxpt ,

We hear you loud and clear. We have dedicated all the resources to get this resolved and it is treated with the outmost urgency. 

I look forward to get you all some good news as soon as possible.

Giac

Meraki Support

[micronet_info]

Perfect timing where our firewalls were pushing a change and now dashboard not available.

Our business is greatly affected by this as firewalls can't replicate with dashboard, we don't have any internet.

We need an ETA and a status page ASAP.

[Techbull]

@GiacomoS 

I assume you may create a list of issues related to this outage so I would like to add that all my SSID's that are tied to a schedule have not been enabled during their scheduled time. However, all the ones that didn't have a schedule were unaffected. 

I'd also like to add a vote to the status page, it would be much more helpful than the community for this type of issue.

[Inderdeep]

@Techbull : I would recommend, Lets list out all issues from everybody here and will check is that related to the hosted services issue. 

[GiacomoS]

Hey @Techbull , 

The SSID availability is pushed from Cloud , so it is going to be impacted by the outage at this moment.

Thank you for adding it here!

Giac

Meraki Support

[JRMcGee]

@itxpt 

I agree!! Our wifi is still down......

[GiacomoS]

Hello @abdunin ,

I hear you and passed the feedback to our Dashboard team. 

Thank you for your patience.

Giac

Meraki Support

[AlphacomItalia]

How long do you think it will be solved?

[itxpt]

From Meraki Support:

Hello Steve

I am really sorry to hear that your attorney's are impacted by this issue, we don't currently have an ETA. I would strongly encourage you to seek an alternative form of having colleagues meet if possible, I am sorry for the inconvenience this has caused you and your team.

Thank you, and I look forward to your response!

...go to the cloud they said, it'll be great...ha!

[abdunin]

This has to be related, a customers MX250 just completely went unreacheable, we could not reach it from the dashboard or ping it by the local LAN IP. We had to reboot it for the internet to come back on at the location and for it to be reachable via LAN. 

Now the light is rainbowing but internet is working. I'm assuming its trying to get the config.

[Kluck]

Would this issue cause client VPN authentication to fail?

[itxpt]

Client VPN is affected.

[blrh]

Since this issue is not resolved, can we remove this from being "Solved"? This is the only public source of information without having to wait an hour+ to get an answer from Meraki Support. 

[GiacomoS]

Thank you for pointing it out @blrh , rectified.

[Jayt]

I'm new to Meraki. Only use AP's but we use 800 of them in our county .gov network.  I'm not impressed with communication so far.

[Inderdeep]

@Jayt : I understand your pain, the issue is its world wide and so many customers are effective, so taking time, Be patience, soon somebody come up to you !

[Jayt]

This is exactly my problem. The SSID's with no schedule are working fine.

"all my SSID's that are tied to a schedule have not been enabled during their scheduled time. However, all the ones that didn't have a schedule were unaffected. "

[Dromero]

all my VPN: tunnel data to a concentrator ssid continue failing

[Maurice_Fellay]

Hi, does this issue also affect Site 2 Site VPN with Meraki appliances on different organizations and Site 2 Site VPN to non meraki appiances?

[JR_NYC]

We have a site to site between two MX84's and the link has been fine.

[Ayala]

Hello, does it affect new devices communicating to the Meraki Cloud Services.  I received my replacement MR33 and its not communicating to the Cloud.  Thank you, much appreciated. 

[Inderdeep]

@Ayala : The issue on the below services as defined earlier as well 

- Device configuration fetch (all devices)

- Client VPN with Meraki Authentication

- Systems Manager enrollment (if gated behind Meraki Authentication)

- SSID scheduling

- Any other service using Meraki Authentication

rest if you have other issue, it may or may not be due to Hosted services

[GiacomoS]

@Maurice_Fellay @Dromero , VPNs on some nodes have also been impacted. Tunnelled SSIDs and Non-Meraki tunnels would be subjected to the outage.

Existing tunnels should continue to operate normally. If they are not, I'd recommend opening a case so we can verify. 

@Ayala , yes, most likely. The MR will not be able to receive the configuration from the Cloud and is therefore unlikely to come online on Dashboard. 

[Maurice_Fellay]

Since the TTL time on site 2 site tunnels on default is 8 hours (28800 secs) will the simple fail if the outages continues over those 8 hours right, because some VPNs are dying while others still work. 

Since when do we have these problems? exactly???? so I can prepare our clients.

Thanks

[collabie14]

That's an excellent question.  I just tried to spin up a new hub and the IP isn't being pushed to spokes because of outage.  Fingers crossed the cloud isn't required for the tunnels to re-negotiate after the TTL expires or a circuit outage.  If not, this is gonna get real interesting real quick. 

[JulienV]

It has been 4 hours now. This was a bad timing for NA East. Can we get an ETA at least?

[amacaya]

We have dropped the VPN for 26 hours.

[TrevorB]

Half a day of billables has been lost for EST. The fact that even a rough window can't be provided so we can inform our clients is ridiculous.

[RazzleDazzle]

Agreed. Also in NA EST. Meraki needs to have a solid ETA and needs to provide a status page moving forward. This is a very poor reflection on Meraki's reliability and service. Not the service expected for the prices Meraki charges. 

[GiacomoS]

@Maurice_Fellay , it would very much be subject to when they rekeyed last.

The issues has surfaced around 2:00 PM BST

Many thanks. 

Giac

Meraki Support

[micronet_info]

@GiacomoS  any updates regarding the resolution or ETA for fix to be implemented?

[amacaya]

- Client VPN with Meraki Authentication, We declared the problem 26 hours ago, case 06834937. A faster solution is required. other organizations begin to be affected.

[GiacomoS]

@amacaya , your original case may not be related to the outage here, but it will likely be impacted throughout today.

Please continue to work with our Support team to continue investigating what the root cause would be. 

Many thanks.

Giac

Meraki Support

[wrespawn]

Throwing in that my SSID is down as well. Last year I was almost sold on switching all our P2P VPN devices with Meraki ones. Decided to renew with what we had, I am so glad I dont have 27 stores offline right now. Just a little bit of WIFI outage isnt brutal for this organization. I will remember this day for years to come next time I get a sales call from Meraki. For sure impacted future business. 

[BigMtnIT]

Starting to get a flood of calls from my customers for Client VPN authentication errors.  Site to Site still working thankfully.  We need this resolved ASAP.  .

[GiacomoS]

Hi all,

We'll start marking updates as "Solutions" so we can have them at the top of the thread. 

Please be on the lookout for them.

Many thanks.

Giac

Meraki Support

[GrahamG]

This is less than reassuring. The first image shows the current dashboard alert message, the second shows the alert message from a couple of hours ago. What happened to "Meraki devices and networks will continue functioning normally "?

Current alert messageAlert message from 8 am PST

[krolunger]

My email and my phone won't stop ringing with questions and anger. Please resolve this promptly. 

[MDR17]

All Meraki customers' Overview map should be showing applicable outage indications (Unavailable SSIDs, Failed Policies, etc.)...  Our Overview map shows ALL GREEN which is NOT correct!  Meraki needs to update the Overview map to reflect correct system statuses for these types of issues.

[JulienV]

VPN looks back for my clients

[GrahamG]

Same here. VPN just connected

[GiacomoS]

Good day everyone,

Our development teams have successfully identified the root cause of the issue and are continuing to work on implementing a fix. 

We understand your frustration and appreciate your continued patience and support during this time. 

[gtatech]

I was able to connect finally and can see now that life may return back to normal.  I hope this never happens again!

[RajuD]

We are able to connect as well.

We have so many questions:

1. Why was there no status page ?

2. Why was no one from Meraki responding on social media about the outage?

3. It took 1 hr 40 mins to get Meraki support on phone just to confirm the issue. It could have been proactively.

4. Why was the change pushed during business hours (if this was caused by change), ? I dont seem to recall any notice about scheduled upgrades.

[srttallhat]

Though I appreciate the PR level of vague updating, We will need an adequate evaluation on what broke, why it broke, and how it can be prevented in the future.  It may be steep, but this seemed to have been a meraki wide outage, and due to the nature of more and more companies relying on remoting services, this 5 hour outage has costed millions in disturbances.

[JRMcGee]

@GiacomoS 

As long time Meraki customer, why did it take 5+ hours to fix this issue. What was the root cause? How is Meraki going to prevent this from happening again? Is there a plan in place to remedy issues like this? This is very concerning that this has happened. 

[Maurice_Fellay]

Site to Site - Non Meraki VPNs are working again.

[MK2]

I'm really surprised, almost every cloud product from cisco now has a status page where even an incident can be tracked transparently for everyone.

Examples?:
https://status.webex.com/service/status?lang=en_US
https://status.umbrella.com/#/

Why don't you have something like that???

[GiacomoS]

@MK2 we hear you and we are passing the feedback on internally. 

Thank you for sharing it!

Giac

[GiacomoS]

Community,

We'd like to thank you all for being so understanding and supportive. We completely understand how difficult the situation has been for many of you. We want you to have a great experience when you use our platform and our products and we are committed to ensure that you do. 

Please be on the lookout for comms via the cases and engage with our Support teams when necessary. 

Keep the feedback coming, it never falls on deaf ears.

Thank you again for all your patience and engagement!

Giac

[NordOps]

One other suggestion. 

This isn't the first time there has been a backend cloud issue and when you call in and wait on hold forever it doesn't help anyone's patience.  It might go a long way to announce an issue like this before putting the call into the support que.  Being able to call Meraki support easily is one of the best features, but being on hold for 45 minutes waiting to talk to someone while back end issues are being worked on by engineers is extremely frustrating.

[SteveB2021]

I second that.

Had the same experience too.

The hold music should be changed to an on hold message stating theirs and outage.

[GiacomoS]

@SteveB2021 @NordOps thank you for sharing!

You are spot on there and I'm happy to share that this is something that we have included in our process. Unfortunately due to some technical difficulties, the announcement was injected after a few hours from the start of the outage and was not immediately played for those who were already in the queue waiting to speak with us. 

Many thanks!

Giac

[r393292492]

Thank you Meraki, my life is now ruined. My wife is walking out the door with the kids, my dog ran away, and my family has disowned me. All I can hear is the constant ringing of my phone and banging on the door of the server closet. I'm scared for my life. Will anyone save me? What will you do to fix this? Will it ever end?

[RazzleDazzle]

Don't forget the angry mob with torches and pitchforks outside your door that is setting fire your home and has brought a vat of boiling Meraki™ oil, just for you. 

[Maurice_Fellay]

Life really sucks when your dog ran away, all other issues you mention are just the most common situation for an IT guy.

But hey, you can always adopt a new dog.

[Maurice_Fellay]

Client VPNs are starting to authenticate, Still changes to dashboard are not being sent to appliances. but at least we are starting to get into normal operation.

Will Meraki send information on what was the issue, why did it happen and what will be the remediation in place so it will not happen again?

[GiacomoS]

@Maurice_Fellay  @srttallhat , please make sure you raise a Dashboard case to request a reason for outage.

Many thanks.

Giac
Meraki Support

[srttallhat]

I have already done so. But transparency in communication to all would help. If I receive an official and adequate response, Ill make sure to post it here.

[GiacomoS]

Hey everyone,

We’re seeing service recovery for the issue impacting certain cloud-based services and working towards full restoration as soon as possible.

We thank our customers and partners for their patience.

[MDR17]

Please post cause of outage and resolution details.

[micronet_info]

@MDR17 my issue is not even fixed and you asking for cause and resolution details? lol

Config still not pushing to devices and showing out of date. 

[srttallhat]

@micronet_info One of many issues stemming from the bigger outage. all in all, an overall official documentation once all is restored is in order, agree?

[Stevesdown]

it's not working for Windows 7. I might add we have a project removing the last ones from our environment. We're using a connection with the extension *.PBK. This method is still not working. 

[SanketKosrabe]

Hello Everyone,

Just to keep you all informed on the config fetch outage/Client VPN, a fix has been deployed to all shards, nodes will now be able to fetch their configurations if a Dashboard change is initiated or so out of date nodes should self-correct as they fetch configs according to their backoff timers.

We thank you for your patience and support on this. 

[DavidHenley]

First, thank you to all for getting this resolved.

But most of us have to answer to someone on what happened today. When will you release a statement on the issues and fixes happening at Meraki to ensure this doesn't happen again?

[amacaya]

I continue with the problem, we have not been able to connect through VPN Client since yesterday, I have case 06834937 open without response, if you can review it again, a capture is attached.

[SanketKosrabe]

@amacaya 

I see that one of the VPN clients was connected on Aug 5 at 13:22:45. If you are still having an issue, I would recommend troubleshooting the issue with support. I see that case is supported pending, NSE will reach back to you soon.

Meanwhile, you can provide a packet capture in Wireshark taken on the MX when the device is trying to connect along with the username from which the connection was being attempted. Also, provide the error core you are seeing on the end device.

I would also recommend testing a VPN connection from another machine such as Mobile/MacOS. 

Make sure, the configurations on the machine is not changed and it is the same as per the document: https://documentation.meraki.com/MX/Client_VPN/Client_VPN_OS_Configuration

- Sanket 

[wrespawn]

Wifi back up, thanks @GiacomoS for the constant updates. Must be a hellova day for you too. 

[DetroitJeremy]

Giacomo,

I appreciate the frequent updates in the thread.  While I have opened a case to request a reason for the outage, I feel that an issue of this magnitude deserves a more official and transparent response.  Root cause and immediate fix information both matter but I believe that future prevention/mitigation is what most of us admins will want to see.  This is what will let us assure our teams, supervisors, and c-levels that this tech stack is still the way to go in our orgs.

-J-

[SanketKosrabe]

@DetroitJeremy @DavidHenley @JRMcGee 

I totally understand the Major issue we had before. We are still working on RCA for the issue and waiting for update from the Engineering team.

Once that verbiage is created, we’ll update all of the cases attached to the issue so our customers get an update regarding RCA. Please look for an email on the cases you have created.

Once again I sincerely thank you for patiently waiting for the issue to be fixed. 

Regards,

Sanket Kosrabe

Cisco Meraki Support 

[BigMtnIT]

Client VPN with Meraki authentication is working again for most of my clients!  Thank you!

[kkwok]

I was talking to support engineer and he has no clue what going on and he wants to start troubleshooting with me. I told him the problem is so i just hang up.

Unbelievable!!

[srttallhat]

@kkwok  In their defense, When something this serious hits so quickly, it requires swift resolution. So understandably the messaage may not have gotten to all the first level techs. But depending on when you spoke with them, if it was hours after, that is rather unacceptable.

[combobulated]

Configs seem to be working for me again.  The symptoms I saw were port config changes (i.e. VLAN changes) showing the correct new settings but port status showing the old ones, and the ports actually behaving somewhere in between.  This caused blocked traffic on port changes that were in progress when this issue hit.  I started to notice some weirdness yesterday afternoon.

[EV1]

While our team continues to work on finalizing the RCA, please feel free to share this summarized report of today's issue:

Today, at 12:00 PM UTC, Meraki dashboard encountered a back end issue that impacted certain cloud-based services. Our Engineering teams have isolated the problem and a fix has been deployed. Teams are actively monitoring the situation. A detailed RCA will follow soon and all impacted customers will be notified once complete. We thank you for your patience.

[tothemoon]

Captive portal authentication is still failing. Any updates on the other issues listed in the original ticket?

[Ryan_Miles]

captive portal should be working. what is your config for the splash page? click through, sponsored, sign-on with (insert your auth provider here), SMS, ISE, SM?

have you opened a Support case?

[tothemoon]

We haven't opened a support case yet. Our configuration is click-through.

[GiacomoS · Accepted solution]

Dear Cisco Meraki Customers and Partners,

On Thursday, August 5, 2021, at 12:00 pm UTC, Cisco Meraki encountered a problem with a certificate expiration that impacted certain cloud-based services, including device configuration, SSID availability, Meraki Authentication, Systems Manager, MV cloud archive, and MT sensor data.

Our engineering teams isolated the problem and renewed the certificate. Services were fully restored at 4:40 pm UTC. Our team is actively working to add additional safeguards to avoid these occurrences in the future

We apologize for any inconvenience this may have caused and thank you for your patience. Should you have any questions or concerns about this update, please contact Meraki support.

Sincerely,
The Cisco Meraki Team

[ACI_Admin]

We are still getting server errors when attempting to register devices in System Mananger.

[JRMcGee]

From CISCO Meraki today............

Dear Cisco Meraki Customer,

On Thursday, August 5, 2021, at 12:00 pm UTC, Cisco Meraki encountered a problem with a certificate expiration that impacted certain cloud-based services, including device configuration, SSID availability, Meraki Authentication, Systems Manager, MV cloud archive, and MT sensor data.

Our engineering teams isolated the problem and renewed the certificate. Services were fully restored at 4:40 pm UTC. Our team is actively working to add additional safeguards to avoid these occurrences in the future

We apologize for any inconvenience this may have caused and thank you for your patience. Should you have any questions or concerns about this update.

Sincerely,
The Cisco Meraki Team

[eyre-jr]

Looks like it probably went out to all customers, but is just a copy & paste of GiacomoS' post above.

FYI - you may want to remove the link in your message, as it exposes your email address/employer identity

[GiacomoS]

@ACI_Admin I'd encourage you to get in touch with our Support team as the issue may be different/unrelated to yesterday's events.

Many thanks!

Giac

[CFMoreu]

Hi to everyone, 

First of all, thanks for implementing the solution, our WFO workers are now able to connect through the customer's VPN with Meraki authentication.

But we have lost more than half a day of work due to this problem.

We are a company with very tight deadlines, so this kind of problem preventing our employees from working without an estimated delivery time is very frustrating.

This has had a direct impact on deliveries to our customers.

In order to maintain confidence in your services and products as Meraki customers, and as @SanketKosrabe has posted here, we request the following:

1. Adequate information from RCA regarding the root cause of this outage, the fix that has been applied.
2. A status page as other Cisco services already have, as other Meraki customers have already requested.
3. A proper style indicator (RED in color) in the control panel with a link to the requested Meraki status page, when a future problem appears (hopefully not).

We have already posted this in our internal case 06842076