meraki APs stopped broadcasting all SSID's suddenly this morning

[RonnieYoyo]

All meraki APs stopped broadcasting SSID suddenly

No issues in the portal.

[rsage_voda]

How are your SSID configured? Tunnelled SSIDs to a concentrator only work if they can reach the Meraki VPN registry

[ww]

https://status.meraki.net/

https://community.meraki.com/t5/Meraki-Service-Notices/Failure-of-configuration-fetch-for-MR/ba-p/21...

[KarstenI]

https://community.meraki.com/t5/Meraki-Service-Notices/Failure-of-configuration-fetch-for-MR/ba-p/21...

[Angel2]

Same issue here, AP's not broadcasting SSID, no problems, no events, no logs showing problems. Two hous ago.

[fattyc_uk]

Complete outage here across 500 users...  If you use availability tags it seems.  Meraki really need to hustle on this one. 

[fattyc_uk]

Can we manage this language on this please Meraki , For some  = ALL APs.  Outage since 07:00 this morning... Nearly 5 hours and 500 users offline... This is a very very serious incident and your language needs to reflect it . 

[RonnieYoyo]

Is a workaround to disable "availability tags" on the APs?

[KarstenI]

Should be fixed very soon.

[fattyc_uk]

5 hours - 500 irate users with NO access . All our services are in the cloud. We have had to send 300 people home.  This is a very , very serious outage for us. We can't cant even provision wired services as we can't update the VLANs on our wired switches due to same issue. 

[fattyc_uk]

Nope - the SSIDs remain firmly bricked . You can't get any updates to the AP . 

[pjc]

Yep just tried disabling SSiD availability - AP's cannot get updated - However, we have other networks in our organisation/dashboard that use SSiD availability schedules and they are broadcasting OK - is this network specific ?

[fattyc_uk]

We have 4  SSIDs with availability schedules , I personally add availability schedules on all SSIDs OOH.  I used to be a tester !!!  Nothing with an availability schedule is being advertised. I have tried standing up temp SSID with no schedule and it never gets advertised... Bricked... 

[RonnieYoyo]

All the sites that are down for us have availability schedules set. Can you see any NTP calls being blocked on the firewall?

Tried turning off availability schedules and setting to 24/7 on but it does not seem to take the update.

[fattyc_uk]

All traffic from AP is fine . No issue there. It's the Meraki online config management . 

[pjc]

Was someone in Meraki trying to fix the ongoing tagging bug (lower case tag names) and broke this ? I hope a full and honest explanation is forthcoming when this has been resolved.

We are looking at removing wired ethernet in the offices and going fully wifi - This sort of outage has got me worrying about that proposed strategy

[fattyc_uk]

If you do use wired use something with a command line where you can effect the outcome.  No switches or APs are manageable at all. There is nothing i can do .  It is a huge issue for Meraki . They have bricked us . 

[WJ1]

Bit of a nightmare this - no APs have received updated config yet despite the update on the fix 1+ hour ago. 

[KarstenI]

Just in case you are not following the Service notices:

"If you are seeing urgent impact on your networks, please proceed to temporarily modify the RF Profile to have the Minimum Bitrate set Per SSID instead of Per band as a workaround, until the issue is fully resolved."

https://community.meraki.com/t5/Meraki-Service-Notices/Failure-of-configuration-fetch-for-MR/bc-p/21...

[RonnieYoyo]

all fixed now

[fattyc_uk]

Cheers Ronnie . What a flipping nightmare.  That is a huge outage for Meraki .  AS a network designer I try never to have my stuff cloud managed but had to in this instance. Lesson learned.. Push back harder... 

[fattyc_uk]

Thanks for all the input !!