cancel
Showing results for 
Search instead for 
Did you mean: 

Major Outage

SOLVED
Kind of a big deal

Major Outage

I am seeing across many shards an outage in the dashboard functionality.  We can't bring up clients or their networks or see their devices.

 

I have contacted support and they have confirmed there is an issue.

1 ACCEPTED SOLUTION

Accepted Solutions
Meraki Employee

Re: Major Outage

After an internal investigation, Cisco Meraki has found the source of the problem which caused an outage of the Meraki dashboard between 9pm and 10:30pm PDT on 8/12. We found that errors in the integration process for dashboard resulted in frontend code being deployed which was no longer compatible with the dashboard application and backend. This issue was resolved and the processes that allowed this issue are currently being improved to prevent problems like this in the future.


Alex S | Cisco Meraki Product Management - Cloud
33 REPLIES 33
Kind of a big deal

Re: Major Outage

I was part way through a site wide switch firmware upgrade.  The whole site has been taken out, and the firmware upgrades have stopped.  Some stack members completed the download, some didn't.

 

Worst possible timing.

Highlighted
Kind of a big deal

Re: Major Outage

1.PNG

Head in the Cloud

Re: Major Outage

Same here @PhilipDAth . I am in the middle of LDAP integration.

 

 

LDAP Outage.PNG

 

 

Cheers
Ajit
ajitsnw@gmail.com

https://www.linkedin.com/in/ajitkumarverma/
Getting noticed

Re: Major Outage

Same here.. 

MK.jpg

Getting noticed

Re: Major Outage

Confirmed here is Australia as well.

 

If it makes you feel any better @PhilipDAth , we were part way commissioning a new site with 50+ cameras, 15+ APs and switches.

 

We have stopped now until this is resolved.

Getting noticed

Re: Major Outage

Did your upgrade take out the dashboard for all of us? =)

Kind of a big deal

Re: Major Outage

>If it makes you feel any better

 

Nope.  :-(

New here

Re: Major Outage

Seeing the same issue in Chicago.

Kind of a big deal

Re: Major Outage

This is obviously a serious global outage affecting either the majority or all shards.

Kind of a big deal

Re: Major Outage

Lets flip this around - anyone in the world still have their dashboard working?

Getting noticed

Re: Major Outage

Yeah ours died too. Funny thing was a team mate was doing his very first Meraki device and of course it died whilst he touched it. So we blamed him for breaking it Smiley Very Happy

Comes here often

Re: Major Outage

nope. our's is not working either.

New here

Re: Major Outage

Makes me feel safe and secure. 

Kind of a big deal

Re: Major Outage

This is actually a security enhancement. 

Getting noticed

Re: Major Outage

We are back online. 

Getting noticed

Re: Major Outage

Yup us too.

Kind of a big deal

Re: Major Outage

Everything looks happy to me as well now.

Building a reputation

Re: Major Outage

Everything up and running (again)

Head in the Cloud

Re: Major Outage

@MeredithW @CarolineS Do you know something about it or can tell us who could tell us about, what the problem was?

 

Seemed like a bigger problem O_o Interessting

Building a reputation

Re: Major Outage

Same chatter on Reddit.

Users on reddit report that, it could look like Meraki outage happened at the same time as a Level3 outage. Meraki might use Level3 somewhere. 

 

Head in the Cloud

Re: Major Outage


@rbnielsen wrote:

Same chatter on Reddit.

Users on reddit report that, it could look like Meraki outage happened at the same time as a Level3 outage. Meraki might use Level3 somewhere. 

 


You mean this post, 2 Years ago? Smiley Wink
27
 
 

Outage this morning?

Anyone else get notified that their AP's went down/up this morning? Ours self-reported an outage from 7:00-7:35 A.M. EDT.

Conversationalist

Re: Major Outage

dashboard is up and running again.

But we don't see the number of networks on the map. Usually we have there some big (mostly) green bubbles showing the number of regional networks.

This is missing, since the management services are up again.

Anybody else seeing this behaviour, too?

Just browsing

Re: Major Outage

Can anyone confirm that the Threat Protection in Security & SD WAN is not working? The page is just loading.

Capture.JPG

 

Conversationalist

Re: Major Outage

confirmed from Germany.
Head in the Cloud

Re: Major Outage


@Axel_R wrote:

dashboard is up and running again.

But we don't see the number of networks on the map. Usually we have there some big (mostly) green bubbles showing the number of regional networks.

This is missing, since the management services are up again.

Anybody else seeing this behaviour, too?


Yes, confirmed.

Building a reputation

Re: Major Outage


@MarcP wrote:

@rbnielsen wrote:

Same chatter on Reddit.

Users on reddit report that, it could look like Meraki outage happened at the same time as a Level3 outage. Meraki might use Level3 somewhere. 

 


You mean this post, 2 Years ago? Smiley Wink
27
 
 

Outage this morning?

Anyone else get notified that their AP's went down/up this morning? Ours self-reported an outage from 7:00-7:35 A.M. EDT.


Meh...
*mumbles something about more coffee...*

Getting noticed

Re: Major Outage

confirmed in UK - just posted another thread about this problem.

 

Also, the Site/Network location markers are missing from the Overview map too Smiley Sad

 

Conversationalist

Re: Major Outage

and our map is zooming always to Africa. Though, we have no networks there...
Case is open with Meraki
Head in the Cloud

Re: Major Outage

As all seems to be fine again, does anyone know what caused the problems?

Meraki Employee

Re: Major Outage

The Meraki cloud engineering and PM team are still investigating this issue. We believe this is resolved now but are still digging into the root cause. I will update this thread later when we have more information.


Alex S | Cisco Meraki Product Management - Cloud
Conversationalist

Re: Major Outage

En México tuvimos la misma afectación se escalo con el soporte de Cisco Meraki y nos comentan lo siguiente.

 

 

 

Saludos,

Gracias por contactar soporte Cisco Meraki!

El problema de visibilidad al portal de Dashboard se debió a problemas en nuestro backend registrados en el día de ayer alrededor de las 21:50 PDT. El problema fue rectificado y los servicios re-establecidos nuevamente.

Favor de notar que durante esta eventualidad los equipos no perdieron funcionalidad, solo el acceso a la plataforma de configuración estaba fuera de servicio.

Si tienes alguna pregunta al respecto no dudes en contactarnos.

 

 

Kind of a big deal

Re: Major Outage

Sounds like yesterday was a good excercise in maintenance mangement for everyone. Hope you all had good plans in place Smiley Tongue

Meraki CMNO, Ruckus WISE, Sonicwall CSSA, Allied Telesis CASE & CAI
Meraki Employee

Re: Major Outage

After an internal investigation, Cisco Meraki has found the source of the problem which caused an outage of the Meraki dashboard between 9pm and 10:30pm PDT on 8/12. We found that errors in the integration process for dashboard resulted in frontend code being deployed which was no longer compatible with the dashboard application and backend. This issue was resolved and the processes that allowed this issue are currently being improved to prevent problems like this in the future.


Alex S | Cisco Meraki Product Management - Cloud
Welcome to the Meraki Community!
To start contributing, simply sign in with your Cisco account. If you don't yet have a Cisco account, you can sign up.