Major Outage

[PhilipDAth]

I am seeing across many shards an outage in the dashboard functionality.  We can't bring up clients or their networks or see their devices.

I have contacted support and they have confirmed there is an issue.

[PhilipDAth]

I was part way through a site wide switch firmware upgrade.  The whole site has been taken out, and the firmware upgrades have stopped.  Some stack members completed the download, some didn't.

Worst possible timing.

[Adoos]

Did your upgrade take out the dashboard for all of us? 😃

[PhilipDAth]

[Adoos]

Same here.. 

[RoryCalhoun]

Seeing the same issue in Chicago.

[AjitKumar]

Same here @PhilipDAth . I am in the middle of LDAP integration.

[Priesty]

Confirmed here is Australia as well.

If it makes you feel any better @PhilipDAth , we were part way commissioning a new site with 50+ cameras, 15+ APs and switches.

We have stopped now until this is resolved.

[PhilipDAth]

>If it makes you feel any better

Nope.  😞

[PhilipDAth]

This is obviously a serious global outage affecting either the majority or all shards.

[PhilipDAth]

Lets flip this around - anyone in the world still have their dashboard working?

[superfly]

Yeah ours died too. Funny thing was a team mate was doing his very first Meraki device and of course it died whilst he touched it. So we blamed him for breaking it 😄

[nubbiess]

nope. our's is not working either.

[RoryCalhoun]

Makes me feel safe and secure. 

[PhilipDAth]

This is actually a security enhancement. 

[Adoos]

We are back online. 

[superfly]

Yup us too.

[PhilipDAth]

Everything looks happy to me as well now.

[CptnCrnch]

Everything up and running (again)

[MarcP]

@MeredithW @CarolineS Do you know something about it or can tell us who could tell us about, what the problem was?

Seemed like a bigger problem O_o Interessting

[Axel_R]

dashboard is up and running again.

But we don't see the number of networks on the map. Usually we have there some big (mostly) green bubbles showing the number of regional networks.

This is missing, since the management services are up again.

Anybody else seeing this behaviour, too?

[MarcP]

---

@Axel_R wrote:

dashboard is up and running again.

But we don't see the number of networks on the map. Usually we have there some big (mostly) green bubbles showing the number of regional networks.

This is missing, since the management services are up again.

Anybody else seeing this behaviour, too?

---

Yes, confirmed.

[rhbirkelund]

Same chatter on Reddit.

Users on reddit report that, it could look like Meraki outage happened at the same time as a Level3 outage. Meraki might use Level3 somewhere. 

[MarcP]

---

@rhbirkelund wrote:

Same chatter on Reddit.

Users on reddit report that, it could look like Meraki outage happened at the same time as a Level3 outage. Meraki might use Level3 somewhere. 

---

You mean this post, 2 Years ago? 😉

27

Posted by

u/jbl0k

2 years ago

Outage this morning?

Anyone else get notified that their AP's went down/up this morning? Ours self-reported an outage from 7:00-7:35 A.M. EDT.

[rhbirkelund]

---

@MarcP wrote:

---

@rhbirkelund wrote:

Same chatter on Reddit.

Users on reddit report that, it could look like Meraki outage happened at the same time as a Level3 outage. Meraki might use Level3 somewhere. 

---

You mean this post, 2 Years ago? 😉

27

Posted by

u/jbl0k

2 years ago

Outage this morning?

Anyone else get notified that their AP's went down/up this morning? Ours self-reported an outage from 7:00-7:35 A.M. EDT.

---

Meh...
*mumbles something about more coffee...*

[Jmbuan10]

Can anyone confirm that the Threat Protection in Security & SD WAN is not working? The page is just loading.

[Axel_R]

confirmed from Germany.

[nealgs]

confirmed in UK - just posted another thread about this problem.

Also, the Site/Network location markers are missing from the Overview map too 😞

[Axel_R]

and our map is zooming always to Africa. Though, we have no networks there...
Case is open with Meraki

[MarcP]

As all seems to be fine again, does anyone know what caused the problems?

[AlexS]

The Meraki cloud engineering and PM team are still investigating this issue. We believe this is resolved now but are still digging into the root cause. I will update this thread later when we have more information.

[AlexS · Accepted solution]

After an internal investigation, Cisco Meraki has found the source of the problem which caused an outage of the Meraki dashboard between 9pm and 10:30pm PDT on 8/12. We found that errors in the integration process for dashboard resulted in frontend code being deployed which was no longer compatible with the dashboard application and backend. This issue was resolved and the processes that allowed this issue are currently being improved to prevent problems like this in the future.

[JBeckner]

Is there some kind of independent Meraki status page to get info on outages like this?

[PhilipDAth]

That would be a great idea!

[CptnCrnch]

What about monitoring with statuspage.io for example?

[JosueHDZ48]

En México tuvimos la misma afectación se escalo con el soporte de Cisco Meraki y nos comentan lo siguiente.

Saludos,

Gracias por contactar soporte Cisco Meraki!

El problema de visibilidad al portal de Dashboard se debió a problemas en nuestro backend registrados en el día de ayer alrededor de las 21:50 PDT. El problema fue rectificado y los servicios re-establecidos nuevamente.

Favor de notar que durante esta eventualidad los equipos no perdieron funcionalidad, solo el acceso a la plataforma de configuración estaba fuera de servicio.

Si tienes alguna pregunta al respecto no dudes en contactarnos.

[BlakeRichardson]

Sounds like yesterday was a good excercise in maintenance mangement for everyone. Hope you all had good plans in place 😛