Hosted Services Issues

SOLVED
GiacomoS
Meraki Employee

Hosted Services Issues

Good day everyone, 

 

We are currently experiencing an issue with multiple hosted services and configuration fetches from Dashboard. 

The impacted services would include:

- Device configuration fetch (all devices)

- Client VPN with Meraki Authentication

- Systems Manager enrollment (if gated behind Meraki Authentication)

- SSID scheduling

- Any other service using Meraki Authentication

- Site to Site VPNs in a few shards

 

Our teams are aware and are working to resolve this as promptly as possible. 

 

The waiting times calling in are currently extended. 

 

We will provide further updates here as we receive them. 

 

Many thanks.

 

Giac

Cisco Meraki Support

Please keep in mind that what I post here is my personal knowledge and opinion. Don't take anything I say for the Holy Grail, but try and see!
Appreciate who helps and be respectful of every opinion and every solution offered.
Share the love, especially the Meraki one!
1 ACCEPTED SOLUTION
GiacomoS
Meraki Employee

Dear Cisco Meraki Customers and Partners,

On Thursday, August 5, 2021, at 12:00 pm UTC, Cisco Meraki encountered a problem with a certificate expiration that impacted certain cloud-based services, including device configuration, SSID availability, Meraki Authentication, Systems Manager, MV cloud archive, and MT sensor data.

Our engineering teams isolated the problem and renewed the certificate. Services were fully restored at 4:40 pm UTC. Our team is actively working to add additional safeguards to avoid these occurrences in the future

We apologize for any inconvenience this may have caused and thank you for your patience. Should you have any questions or concerns about this update, please contact Meraki support.

Sincerely,
The Cisco Meraki Team

Please keep in mind that what I post here is my personal knowledge and opinion. Don't take anything I say for the Holy Grail, but try and see!
Appreciate who helps and be respectful of every opinion and every solution offered.
Share the love, especially the Meraki one!

View solution in original post

94 REPLIES 94
Jayt
New here

Once this gets sorted out, I'd like to know why this is effecting our local SSID clients from connecting via our local NAT network.  New to Meraki.

 

Thanks

 

sixgigdan
New here

Do we have any kind of ETA on successful resolution of this issue?

 

Also.....WHY DON'T YOU HAVE A STATUS PAGE?!

Agreed. Such a simple thing as most all cloud providers have. 

 

Hey @sixgigdan , 

 

Not yet unfortunately, but we have multiple teams assisting to resolve this as soon as possible. 

 

A root cause has been identified and we are currently working on a fix.

 

I hear your feedback with the Status page and will ensure I raise it up.

 

Many thanks for your patience everyone.

 

Giac

Meraki Support

Please keep in mind that what I post here is my personal knowledge and opinion. Don't take anything I say for the Holy Grail, but try and see!
Appreciate who helps and be respectful of every opinion and every solution offered.
Share the love, especially the Meraki one!

Not having an ETA for essentially a full stop of service is incredibly concerning. Some businesses have no other recourse in accessing local resources in areas with a WFH order. There are no status updates, and nothing to indicate progress aside from a green banner at the top of the cloud portal. I have nothing to provide our team in regards to when to expect service restoration.

We have a venue with more than 1000 guest with no internet connection and can't take any payments as we don't have any internet.


2x Meraki MX 250 and 8 x Meraki MS250-48LP along with 20+ AP's are all useless as firewalls are out of config and we can't bring them online (firewall rules). 

 

 

@GiacomoS We need an ETA and get this fixed as soon as possible.

@micronet_info : Meraki is all together working on the issue on high priority and will soon update with the ETA @GiacomoS  

Regards
Inderdeep Singh
www.thenetworkdna.com ( Awarded by Cisco IT Blogs award 2020)
abdunin
Conversationalist

And why is the alert for this outage on the Meraki dashboard in green? That typically indicates a non-critical message. We're currently unable to configure our firewalls, config out of date for the last 45 mins.

This is taking way too long, for a top tier support company this should have been fixed by now, lots of folks riding on this and are now dead in the water.

GiacomoS
Meraki Employee

Hey @itxpt ,

 

We hear you loud and clear. We have dedicated all the resources to get this resolved and it is treated with the outmost urgency. 

 

I look forward to get you all some good news as soon as possible.

 

Giac

Meraki Support

 

Please keep in mind that what I post here is my personal knowledge and opinion. Don't take anything I say for the Holy Grail, but try and see!
Appreciate who helps and be respectful of every opinion and every solution offered.
Share the love, especially the Meraki one!

Perfect timing where our firewalls were pushing a change and now dashboard not available.

 

Our business is greatly affected by this as firewalls can't replicate with dashboard, we don't have any internet.

 

We need an ETA and a status page ASAP.

Techbull
Conversationalist

@GiacomoS 

I assume you may create a list of issues related to this outage so I would like to add that all my SSID's that are tied to a schedule have not been enabled during their scheduled time. However, all the ones that didn't have a schedule were unaffected. 

I'd also like to add a vote to the status page, it would be much more helpful than the community for this type of issue.

Inderdeep
Kind of a big deal

@Techbull : I would recommend, Lets list out all issues from everybody here and will check is that related to the hosted services issue. 

Regards
Inderdeep Singh
www.thenetworkdna.com ( Awarded by Cisco IT Blogs award 2020)

Hey @Techbull , 

The SSID availability is pushed from Cloud , so it is going to be impacted by the outage at this moment.

 

Thank you for adding it here!

 

Giac

Meraki Support

Please keep in mind that what I post here is my personal knowledge and opinion. Don't take anything I say for the Holy Grail, but try and see!
Appreciate who helps and be respectful of every opinion and every solution offered.
Share the love, especially the Meraki one!
JRMcGee
Conversationalist

@itxpt 

I agree!! Our wifi is still down......

Hello @abdunin ,

 

I hear you and passed the feedback to our Dashboard team. 

 

Thank you for your patience.

 

Giac

Meraki Support

Please keep in mind that what I post here is my personal knowledge and opinion. Don't take anything I say for the Holy Grail, but try and see!
Appreciate who helps and be respectful of every opinion and every solution offered.
Share the love, especially the Meraki one!
AlphacomItalia
Here to help

How long do you think it will be solved?

From Meraki Support:

 

Hello Steve

I am really sorry to hear that your attorney's are impacted by this issue, we don't currently have an ETA. I would strongly encourage you to seek an alternative form of having colleagues meet if possible, I am sorry for the inconvenience this has caused you and your team.

Thank you, and I look forward to your response!

 

...go to the cloud they said, it'll be great...ha!

abdunin
Conversationalist

This has to be related, a customers MX250 just completely went unreacheable, we could not reach it from the dashboard or ping it by the local LAN IP. We had to reboot it for the internet to come back on at the location and for it to be reachable via LAN. 

 

Now the light is rainbowing but internet is working. I'm assuming its trying to get the config.

Kluck
New here

Would this issue cause client VPN authentication to fail?

Client VPN is affected.

blrh
New here

Since this issue is not resolved, can we remove this from being "Solved"? This is the only public source of information without having to wait an hour+ to get an answer from Meraki Support. 

GiacomoS
Meraki Employee

Thank you for pointing it out @blrh , rectified.

 

 

Please keep in mind that what I post here is my personal knowledge and opinion. Don't take anything I say for the Holy Grail, but try and see!
Appreciate who helps and be respectful of every opinion and every solution offered.
Share the love, especially the Meraki one!
Jayt
New here

I'm new to Meraki. Only use AP's but we use 800 of them in our county .gov network.  I'm not impressed with communication so far.

Inderdeep
Kind of a big deal

@Jayt : I understand your pain, the issue is its world wide and so many customers are effective, so taking time, Be patience, soon somebody come up to you !

Regards
Inderdeep Singh
www.thenetworkdna.com ( Awarded by Cisco IT Blogs award 2020)
Jayt
New here

This is exactly my problem. The SSID's with no schedule are working fine.

 

"all my SSID's that are tied to a schedule have not been enabled during their scheduled time. However, all the ones that didn't have a schedule were unaffected. "

Dromero
Here to help

all my VPN: tunnel data to a concentrator ssid continue failing

 

 

Maurice_Fellay
Conversationalist

Hi, does this issue also affect Site 2 Site VPN with Meraki appliances on different organizations and Site 2 Site VPN to non meraki appiances?

We have a site to site between two MX84's and the link has been fine.

Ayala
Conversationalist

Hello, does it affect new devices communicating to the Meraki Cloud Services.  I received my replacement MR33 and its not communicating to the Cloud.  Thank you, much appreciated. 

Inderdeep
Kind of a big deal

@Ayala : The issue on the below services as defined earlier as well 

- Device configuration fetch (all devices)

- Client VPN with Meraki Authentication

- Systems Manager enrollment (if gated behind Meraki Authentication)

- SSID scheduling

- Any other service using Meraki Authentication

 

rest if you have other issue, it may or may not be due to Hosted services

Regards
Inderdeep Singh
www.thenetworkdna.com ( Awarded by Cisco IT Blogs award 2020)
GiacomoS
Meraki Employee

@Maurice_Fellay @Dromero , VPNs on some nodes have also been impacted. Tunnelled SSIDs and Non-Meraki tunnels would be subjected to the outage.

 

Existing tunnels should continue to operate normally. If they are not, I'd recommend opening a case so we can verify. 

 

@Ayala , yes, most likely. The MR will not be able to receive the configuration from the Cloud and is therefore unlikely to come online on Dashboard. 

Please keep in mind that what I post here is my personal knowledge and opinion. Don't take anything I say for the Holy Grail, but try and see!
Appreciate who helps and be respectful of every opinion and every solution offered.
Share the love, especially the Meraki one!

Since the TTL time on site 2 site tunnels on default is 8 hours (28800 secs) will the simple fail if the outages continues over those 8 hours right, because some VPNs are dying while others still work. 

 

Since when do we have these problems? exactly???? so I can prepare our clients.

 

Thanks

That's an excellent question.  I just tried to spin up a new hub and the IP isn't being pushed to spokes because of outage.  Fingers crossed the cloud isn't required for the tunnels to re-negotiate after the TTL expires or a circuit outage.  If not, this is gonna get real interesting real quick. 

It has been 4 hours now. This was a bad timing for NA East. Can we get an ETA at least?

We have dropped the VPN for 26 hours.

Half a day of billables has been lost for EST. The fact that even a rough window can't be provided so we can inform our clients is ridiculous.

Agreed. Also in NA EST. Meraki needs to have a solid ETA and needs to provide a status page moving forward. This is a very poor reflection on Meraki's reliability and service. Not the service expected for the prices Meraki charges. 

@Maurice_Fellay , it would very much be subject to when they rekeyed last.

 

The issues has surfaced around 2:00 PM BST

 

Many thanks. 

 

Giac

Meraki Support

Please keep in mind that what I post here is my personal knowledge and opinion. Don't take anything I say for the Holy Grail, but try and see!
Appreciate who helps and be respectful of every opinion and every solution offered.
Share the love, especially the Meraki one!

@GiacomoS  any updates regarding the resolution or ETA for fix to be implemented?

 

 

amacaya
New here

- Client VPN with Meraki Authentication, We declared the problem 26 hours ago, case 06834937. A faster solution is required. other organizations begin to be affected.

@amacaya , your original case may not be related to the outage here, but it will likely be impacted throughout today.

 

Please continue to work with our Support team to continue investigating what the root cause would be. 

 

Many thanks.

 

Giac

Meraki Support

Please keep in mind that what I post here is my personal knowledge and opinion. Don't take anything I say for the Holy Grail, but try and see!
Appreciate who helps and be respectful of every opinion and every solution offered.
Share the love, especially the Meraki one!
wrespawn
Here to help

Throwing in that my SSID is down as well. Last year I was almost sold on switching all our P2P VPN devices with Meraki ones. Decided to renew with what we had, I am so glad I dont have 27 stores offline right now. Just a little bit of WIFI outage isnt brutal for this organization. I will remember this day for years to come next time I get a sales call from Meraki. For sure impacted future business. 

BigMtnIT
New here

Starting to get a flood of calls from my customers for Client VPN authentication errors.  Site to Site still working thankfully.  We need this resolved ASAP.  .

GiacomoS
Meraki Employee

Hi all,

 

We'll start marking updates as "Solutions" so we can have them at the top of the thread. 

 

Please be on the lookout for them.

 

Many thanks.


Giac

Meraki Support

Please keep in mind that what I post here is my personal knowledge and opinion. Don't take anything I say for the Holy Grail, but try and see!
Appreciate who helps and be respectful of every opinion and every solution offered.
Share the love, especially the Meraki one!
GrahamG
Here to help

This is less than reassuring. The first image shows the current dashboard alert message, the second shows the alert message from a couple of hours ago. What happened to "Meraki devices and networks will continue functioning normally "?

Current alert messageCurrent alert messageAlert message from 8 am PSTAlert message from 8 am PST

krolunger
New here

My email and my phone won't stop ringing with questions and anger. Please resolve this promptly. 

MDR17
Conversationalist

All Meraki customers' Overview map should be showing applicable outage indications (Unavailable SSIDs, Failed Policies, etc.)...  Our Overview map shows ALL GREEN which is NOT correct!  Meraki needs to update the Overview map to reflect correct system statuses for these types of issues.

VPN looks back for my clients

Same here. VPN just connected

GiacomoS
Meraki Employee

Good day everyone,

 

Our development teams have successfully identified the root cause of the issue and are continuing to work on implementing a fix. 

 

We understand your frustration and appreciate your continued patience and support during this time. 

 

 

Please keep in mind that what I post here is my personal knowledge and opinion. Don't take anything I say for the Holy Grail, but try and see!
Appreciate who helps and be respectful of every opinion and every solution offered.
Share the love, especially the Meraki one!
gtatech
Conversationalist

I was able to connect finally and can see now that life may return back to normal.  I hope this never happens again!

We are able to connect as well.

 

We have so many questions:

1. Why was there no status page ?

2. Why was no one from Meraki responding on social media about the outage?

3. It took 1 hr 40 mins to get Meraki support on phone just to confirm the issue. It could have been proactively.

4. Why was the change pushed during business hours (if this was caused by change), ? I dont seem to recall any notice about scheduled upgrades.

 

 

Though I appreciate the PR level of vague updating, We will need an adequate evaluation on what broke, why it broke, and how it can be prevented in the future.  It may be steep, but this seemed to have been a meraki wide outage, and due to the nature of more and more companies relying on remoting services, this 5 hour outage has costed millions in disturbances.

 

As long time Meraki customer, why did it take 5+ hours to fix this issue. What was the root cause? How is Meraki going to prevent this from happening again? Is there a plan in place to remedy issues like this? This is very concerning that this has happened. 

Site to Site - Non Meraki VPNs are working again.

I'm really surprised, almost every cloud product from cisco now has a status page where even an incident can be tracked transparently for everyone.

Examples?:
https://status.webex.com/service/status?lang=en_US
https://status.umbrella.com/#/


Why don't you have something like that???

GiacomoS
Meraki Employee

@MK2 we hear you and we are passing the feedback on internally. 

 

Thank you for sharing it!

Giac

Please keep in mind that what I post here is my personal knowledge and opinion. Don't take anything I say for the Holy Grail, but try and see!
Appreciate who helps and be respectful of every opinion and every solution offered.
Share the love, especially the Meraki one!

Community,

 

We'd like to thank you all for being so understanding and supportive. We completely understand how difficult the situation has been for many of you. We want you to have a great experience when you use our platform and our products and we are committed to ensure that you do. 

 

Please be on the lookout for comms via the cases and engage with our Support teams when necessary. 

 

Keep the feedback coming, it never falls on deaf ears.

 

Thank you again for all your patience and engagement!

 

Giac

Please keep in mind that what I post here is my personal knowledge and opinion. Don't take anything I say for the Holy Grail, but try and see!
Appreciate who helps and be respectful of every opinion and every solution offered.
Share the love, especially the Meraki one!

One other suggestion. 

 

This isn't the first time there has been a backend cloud issue and when you call in and wait on hold forever it doesn't help anyone's patience.  It might go a long way to announce an issue like this before putting the call into the support que.  Being able to call Meraki support easily is one of the best features, but being on hold for 45 minutes waiting to talk to someone while back end issues are being worked on by engineers is extremely frustrating.

I second that.

Had the same experience too.

The hold music should be changed to an on hold message stating theirs and outage.

@SteveB2021 @NordOps thank you for sharing!

 

You are spot on there and I'm happy to share that this is something that we have included in our process. Unfortunately due to some technical difficulties, the announcement was injected after a few hours from the start of the outage and was not immediately played for those who were already in the queue waiting to speak with us. 

 

Many thanks!

Giac

Please keep in mind that what I post here is my personal knowledge and opinion. Don't take anything I say for the Holy Grail, but try and see!
Appreciate who helps and be respectful of every opinion and every solution offered.
Share the love, especially the Meraki one!
r393292492
Conversationalist

Thank you Meraki, my life is now ruined. My wife is walking out the door with the kids, my dog ran away, and my family has disowned me. All I can hear is the constant ringing of my phone and banging on the door of the server closet. I'm scared for my life. Will anyone save me? What will you do to fix this? Will it ever end?

Don't forget the angry mob with torches and pitchforks outside your door that is setting fire your home and has brought a vat of boiling Meraki™ oil, just for you. 

Life really sucks when your dog ran away, all other issues you mention are just the most common situation for an IT guy.

 

But hey, you can always adopt a new dog.

Maurice_Fellay
Conversationalist

Client VPNs are starting to authenticate, Still changes to dashboard are not being sent to appliances. but at least we are starting to get into normal operation.

 

Will Meraki send information on what was the issue, why did it happen and what will be the remediation in place so it will not happen again?

@Maurice_Fellay  @srttallhat , please make sure you raise a Dashboard case to request a reason for outage.

 

Many thanks.

 

Giac
Meraki Support

Please keep in mind that what I post here is my personal knowledge and opinion. Don't take anything I say for the Holy Grail, but try and see!
Appreciate who helps and be respectful of every opinion and every solution offered.
Share the love, especially the Meraki one!

I have already done so. But transparency in communication to all would help. If I receive an official and adequate response, Ill make sure to post it here.

Hey everyone,

 

We’re seeing service recovery for the issue impacting certain cloud-based services and working towards full restoration as soon as possible.

 

We thank our customers and partners for their patience.

Please keep in mind that what I post here is my personal knowledge and opinion. Don't take anything I say for the Holy Grail, but try and see!
Appreciate who helps and be respectful of every opinion and every solution offered.
Share the love, especially the Meraki one!
MDR17
Conversationalist

Please post cause of outage and resolution details.

@MDR17 my issue is not even fixed and you asking for cause and resolution details? lol

 

Config still not pushing to devices and showing out of date. 

@micronet_info One of many issues stemming from the bigger outage. all in all, an overall official documentation once all is restored is in order, agree?