Major Outage

SOLVED
PhilipDAth
Kind of a big deal
Kind of a big deal

Major Outage

I am seeing across many shards an outage in the dashboard functionality.  We can't bring up clients or their networks or see their devices.

 

I have contacted support and they have confirmed there is an issue.

1 ACCEPTED SOLUTION
AlexS
Meraki Employee
Meraki Employee

After an internal investigation, Cisco Meraki has found the source of the problem which caused an outage of the Meraki dashboard between 9pm and 10:30pm PDT on 8/12. We found that errors in the integration process for dashboard resulted in frontend code being deployed which was no longer compatible with the dashboard application and backend. This issue was resolved and the processes that allowed this issue are currently being improved to prevent problems like this in the future.


Alex S | Cisco Meraki Product Management - Cloud

View solution in original post

36 REPLIES 36
PhilipDAth
Kind of a big deal
Kind of a big deal

I was part way through a site wide switch firmware upgrade.  The whole site has been taken out, and the firmware upgrades have stopped.  Some stack members completed the download, some didn't.

 

Worst possible timing.

Adoos
Building a reputation

Did your upgrade take out the dashboard for all of us? 😃

PhilipDAth
Kind of a big deal
Kind of a big deal

1.PNG

Adoos
Building a reputation

Same here.. 

MK.jpg

Seeing the same issue in Chicago.

AjitKumar
Head in the Cloud

Same here @PhilipDAth . I am in the middle of LDAP integration.

 

 

LDAP Outage.PNG

 

 

Regards,
Ajit
AjitsNW@gmail.com
www.ajit.network
Priesty
Building a reputation

Confirmed here is Australia as well.

 

If it makes you feel any better @PhilipDAth , we were part way commissioning a new site with 50+ cameras, 15+ APs and switches.

 

We have stopped now until this is resolved.

PhilipDAth
Kind of a big deal
Kind of a big deal

>If it makes you feel any better

 

Nope.  😞

This is obviously a serious global outage affecting either the majority or all shards.

Lets flip this around - anyone in the world still have their dashboard working?

Yeah ours died too. Funny thing was a team mate was doing his very first Meraki device and of course it died whilst he touched it. So we blamed him for breaking it 😄

nope. our's is not working either.

Makes me feel safe and secure. 

PhilipDAth
Kind of a big deal
Kind of a big deal

This is actually a security enhancement. 

Adoos
Building a reputation

We are back online. 

superfly
Getting noticed

Yup us too.

PhilipDAth
Kind of a big deal
Kind of a big deal

Everything looks happy to me as well now.

Everything up and running (again)

MarcP
Kind of a big deal

@MeredithW @CarolineS Do you know something about it or can tell us who could tell us about, what the problem was?

 

Seemed like a bigger problem O_o Interessting

dashboard is up and running again.

But we don't see the number of networks on the map. Usually we have there some big (mostly) green bubbles showing the number of regional networks.

This is missing, since the management services are up again.

Anybody else seeing this behaviour, too?

MarcP
Kind of a big deal


@Axel_R wrote:

dashboard is up and running again.

But we don't see the number of networks on the map. Usually we have there some big (mostly) green bubbles showing the number of regional networks.

This is missing, since the management services are up again.

Anybody else seeing this behaviour, too?


Yes, confirmed.

rhbirkelund
Kind of a big deal

Same chatter on Reddit.

Users on reddit report that, it could look like Meraki outage happened at the same time as a Level3 outage. Meraki might use Level3 somewhere. 

 

LinkedIn ::: https://blog.rhbirkelund.dk/

Like what you see? - Give a Kudo ## Did it answer your question? - Mark it as a Solution 🙂

All code examples are provided as is. Responsibility for Code execution lies solely your own.
MarcP
Kind of a big deal


@rhbirkelund wrote:

Same chatter on Reddit.

Users on reddit report that, it could look like Meraki outage happened at the same time as a Level3 outage. Meraki might use Level3 somewhere. 

 


You mean this post, 2 Years ago? 😉
27
 
 

Outage this morning?

Anyone else get notified that their AP's went down/up this morning? Ours self-reported an outage from 7:00-7:35 A.M. EDT.

rhbirkelund
Kind of a big deal


@MarcP wrote:

@rhbirkelund wrote:

Same chatter on Reddit.

Users on reddit report that, it could look like Meraki outage happened at the same time as a Level3 outage. Meraki might use Level3 somewhere. 

 


You mean this post, 2 Years ago? 😉
27
 
 

Outage this morning?

Anyone else get notified that their AP's went down/up this morning? Ours self-reported an outage from 7:00-7:35 A.M. EDT.


Meh...
*mumbles something about more coffee...*

LinkedIn ::: https://blog.rhbirkelund.dk/

Like what you see? - Give a Kudo ## Did it answer your question? - Mark it as a Solution 🙂

All code examples are provided as is. Responsibility for Code execution lies solely your own.
Jmbuan10
Comes here often

Can anyone confirm that the Threat Protection in Security & SD WAN is not working? The page is just loading.

Capture.JPG

 

confirmed from Germany.
nealgs
Building a reputation

confirmed in UK - just posted another thread about this problem.

 

Also, the Site/Network location markers are missing from the Overview map too 😞

 

and our map is zooming always to Africa. Though, we have no networks there...
Case is open with Meraki
MarcP
Kind of a big deal

As all seems to be fine again, does anyone know what caused the problems?

AlexS
Meraki Employee
Meraki Employee

The Meraki cloud engineering and PM team are still investigating this issue. We believe this is resolved now but are still digging into the root cause. I will update this thread later when we have more information.


Alex S | Cisco Meraki Product Management - Cloud
AlexS
Meraki Employee
Meraki Employee

After an internal investigation, Cisco Meraki has found the source of the problem which caused an outage of the Meraki dashboard between 9pm and 10:30pm PDT on 8/12. We found that errors in the integration process for dashboard resulted in frontend code being deployed which was no longer compatible with the dashboard application and backend. This issue was resolved and the processes that allowed this issue are currently being improved to prevent problems like this in the future.


Alex S | Cisco Meraki Product Management - Cloud
JBeckner
Conversationalist

Is there some kind of independent Meraki status page to get info on outages like this?

 

 

PhilipDAth
Kind of a big deal
Kind of a big deal

That would be a great idea!

What about monitoring with statuspage.io for example?

JosueHDZ48
Conversationalist

En México tuvimos la misma afectación se escalo con el soporte de Cisco Meraki y nos comentan lo siguiente.

 

 

 

Saludos,

Gracias por contactar soporte Cisco Meraki!

El problema de visibilidad al portal de Dashboard se debió a problemas en nuestro backend registrados en el día de ayer alrededor de las 21:50 PDT. El problema fue rectificado y los servicios re-establecidos nuevamente.

Favor de notar que durante esta eventualidad los equipos no perdieron funcionalidad, solo el acceso a la plataforma de configuración estaba fuera de servicio.

Si tienes alguna pregunta al respecto no dudes en contactarnos.

 

 

Sounds like yesterday was a good excercise in maintenance mangement for everyone. Hope you all had good plans in place 😛

Get notified when there are additional replies to this discussion.