Switch SNMPv3 settings changed to AES128 without warning

[MikeHunt]

One of my sites's SNMPv3 reporting went offline earlier tonight.

The Sites' > Network-Wide > Configure > General > SNMP section had a new Privacy Mode = AES128 dropdown.

Of course our monitoring software is still configured for the previous DES setting

This is easy enough to 'update' our Network Monitor to AES too, which is definitely the preferred option - but seriously WTF was this changed to begin with? This wasn't a change we triggered nor did I receive any advice about this. This caused mass panic & confusion as to why we're suddenly started loosing SNMP polling progressively across all Meraki switches on the site.


To possibly make matters worse - Some of my other sites also have this new AES128 dropdown - yet are still polling fine on the existing DES settings.  WTF is going on with the change control process here?

[alemabrahao]

It's a complicated situation and I'm sorry for you, but it would be a good idea for you to contact Meraki support.

[matts3]

Yes! I just noticed this too, all of our devices just dropped off from SNMP monitoring and I couldn't work out why. Thank you for sharing this.

It's worth noting that if you have mixed networks, i.e. both switching and wireless, that drop down doesn't even show up!

Annoyingly, this change hasn't hit all of our networks yet, just the one or two :S

Thanks Meraki for making silent changes....

[elves]

having the same issue now random devices area having communication issues on prtg and others dont

[RossFawcett]

Can confirm we are seeing the same issue here, switches all started dropping off randomly, swapping to AES restores them, but have some still working with DES or AES. And as others have noted, in combined networks we don't even see the option to select AES.

We have a case open with Meraki now.

[MikeHunt]

Further to this original incident, two more of more sites had some of their switches switches also suffer the same fate.

I can also confirm the 'combined network' sites do not sure the DES/AES selection. HOWEVER upon pressing 'save' on that config page - it then changes all site's switches to use AES! 😲

So I've now re-saved all my sites to force them to AES - which let's be honest, should've been the default setting all along for a switch at this price point.

[matts3]

Agreed, I raised a case with them a while back complaining about the lack of AES on switches, perhaps they've actually implemented my feature request without telling anyone 😄

Though, the ones I 'fixed' yesterday by changing to AES, are now only working on DES.... coming on Meraki/Cisco, what's going on!

I've also noticed our 802.1x on swtiches have broken as well (ticket raised), so god knows what else they've silently changed too!

[matts3]

Whilst writing my previous response, even more switches have gone back to DES........ 😐 - Mind blown!

[MikeHunt]

AND the same thing has happening for me too for the same site that dropped about this time yesterday.

Back to DES now

AND now SNMP uptimes have been reset too - yet my network monitoring didn't show a network drop at all? In fact I saw nothing go offline. But this is royally screwing my metrics and reporting!!!

[matts3]

Support have told me this:

"To clarify why you witnessed a change between AES and DES is due to a backend maintenance change was introduced which caused this issue, which caused your organisation configuration which was set to AES to override your network settings which was set to DES. It was reverted back which is why it went from AES back to DES. I understand this can be quite confusing so I hope my explanation above has addressed your concerns."

I've asked them to give me AES back, on all my sites haha.