Switch SNMPv3 settings changed to AES128 without warning

[MikeHunt]

One of my sites's SNMPv3 reporting went offline earlier tonight.

The Sites' > Network-Wide > Configure > General > SNMP section had a new Privacy Mode = AES128 dropdown.

Of course our monitoring software is still configured for the previous DES setting

This is easy enough to 'update' our Network Monitor to AES too, which is definitely the preferred option - but seriously WTF was this changed to begin with? This wasn't a change we triggered nor did I receive any advice about this. This caused mass panic & confusion as to why we're suddenly started loosing SNMP polling progressively across all Meraki switches on the site.


To possibly make matters worse - Some of my other sites also have this new AES128 dropdown - yet are still polling fine on the existing DES settings.  WTF is going on with the change control process here?

[alemabrahao]

It's a complicated situation and I'm sorry for you, but it would be a good idea for you to contact Meraki support.

[matts3]

Yes! I just noticed this too, all of our devices just dropped off from SNMP monitoring and I couldn't work out why. Thank you for sharing this.

It's worth noting that if you have mixed networks, i.e. both switching and wireless, that drop down doesn't even show up!

Annoyingly, this change hasn't hit all of our networks yet, just the one or two :S

Thanks Meraki for making silent changes....

[elves]

having the same issue now random devices area having communication issues on prtg and others dont

[RossFawcett]

Can confirm we are seeing the same issue here, switches all started dropping off randomly, swapping to AES restores them, but have some still working with DES or AES. And as others have noted, in combined networks we don't even see the option to select AES.

We have a case open with Meraki now.

[MikeHunt]

Further to this original incident, two more of more sites had some of their switches switches also suffer the same fate.

I can also confirm the 'combined network' sites do not sure the DES/AES selection. HOWEVER upon pressing 'save' on that config page - it then changes all site's switches to use AES! 😲

So I've now re-saved all my sites to force them to AES - which let's be honest, should've been the default setting all along for a switch at this price point.

[matts3]

Agreed, I raised a case with them a while back complaining about the lack of AES on switches, perhaps they've actually implemented my feature request without telling anyone 😄

Though, the ones I 'fixed' yesterday by changing to AES, are now only working on DES.... coming on Meraki/Cisco, what's going on!

I've also noticed our 802.1x on swtiches have broken as well (ticket raised), so god knows what else they've silently changed too!

[matts3]

Whilst writing my previous response, even more switches have gone back to DES........ 😐 - Mind blown!

[MikeHunt]

AND the same thing has happening for me too for the same site that dropped about this time yesterday.

Back to DES now

AND now SNMP uptimes have been reset too - yet my network monitoring didn't show a network drop at all? In fact I saw nothing go offline. But this is royally screwing my metrics and reporting!!!

[matts3]

Support have told me this:

"To clarify why you witnessed a change between AES and DES is due to a backend maintenance change was introduced which caused this issue, which caused your organisation configuration which was set to AES to override your network settings which was set to DES. It was reverted back which is why it went from AES back to DES. I understand this can be quite confusing so I hope my explanation above has addressed your concerns."

I've asked them to give me AES back, on all my sites haha.

[RossFawcett]

We didn't even get that

Greetings,

Thank you for reaching out to Cisco Meraki Technical Support.

You may be encountering an ongoing known issue affecting SNMPv3 which is under active investigation internally.

As part of testing, please change the Organization > Settings >  SNMP privacy mode to DES and verify if SNMP begins responding as normal once again.

Please let me know if you have any questions.

---

By this point we'd already gone in, hit save on all the networks and moved over to AES. Come today, now the same switches which were fixed by moving to AES, are a mixed bag if they work on DES or AES now... Tried the classic, turn it off and on for SNMP, re-save the settings at the network and org level, and still have some devices that are not functional on AES yet are configured to use AES..

What a mess.

[RossFawcett]

We have now ended up down the same path. We'd get rolling SNMP down alerts from devices that were working fine for a bit on AES, then would suddenly stop, you'd test them and they were working on DES again. It's almost like there is something competing to update the configuration, causing it to flip back and forth between AES and DES. DES so far seems to be more stable, but only time will tell.

[GiacomoS]

Hey folks,

We are looking into it right now with our development teams.

Please bear with us; if you don't have a case open, can I please ask you to raise one, so we can keep you appraised with the updates?

Many thanks!

Giac

[GiacomoS]

Hey Community,

I believe this should be resolved now. If you are still seeing more impact, please do let us know via the Support case so we can continue investigating.

Many thanks!

Giac

[RossFawcett]

It was still broken/inconsistent as of a few minutes ago prior to us rolling everything back to DES. When exactly was the fix rolled out?

[RossFawcett]

I can with confidence say it is not resolved. Swapping back to AES, immediately same problems.

Network and org are all set to AES, switch config reports up to date, but 8 out of the 13 switches in that location do not respond on AES and only work on DES..

[matts3]

What do you mean by resolved.... What have you "resolved" exactly?

Have you turned AES back on for us?

Ideally, it would be nice to be able to pick between DES and AES for the switches and the access points... 

At the moment, I'm still seeing everything on DES.

[GiacomoS]

Hey folks, 
My understanding is that where there was a conflict between the org settings and the local settings, the local conflict got overwritten. The configuration should have restored to whatever you have set up on Dashboard.

That is the part that was resolved; you'll still need to ensure that the encryption is aligned between the Dashboard configuration and the SNMP server. 

I'm not quite sure about the AES availability at the moment, as that was somewhat beyond the scope of the bug that was identified. 

Again, if you are seeing any other bizarre behaviour, please ensure you relay it to our Support engineers, so we can have a proper look. 

Many thanks!

Giac

[matts3]

Thanks Giac,

What I find confusing is that in the Org settings, you can enable snmpv3 and provide credentials, and set an encryption layer, but this bares no relevance to the network-wide configurations, where we can't even select what type of encryption we want...

I appreciate you reaching out to us via the forum.

[GiacomoS]

Hey @matts3 ,

Yeah I think that was the root of the problem 😅 The org wide settings should not have had any bearing, and that was the unexpected behaviour we should have squashed. 

Thank you for your patience with this!

Giac

[matts3]

Okay great, so when can we get AES back on our switches for SNMP encryption, because well... AES has been around since 2001, and should be the industry standard.

As we've seen AES can be deployed to the switches for SNMP, why can't we use it??

[MikeHunt]

I've just had another 6 switches drop from AES auth now too.

[helpdeskdan]

Opened a case yesterday, no response as of yet.  I resorted to write some python code that queries SNMP via DES and AES and updates sql in LibreNMS accordingly. 

I did not see the org settings for SNMP somehow changed to AES.  I'll set that to DES and see if this stops the random changes. Thanks.

[StevenBeasley]

Way to go!

Great job.

[rc0123738]

I am quite hesitant to make a change. PRTG is listed as DES, and Meraki is listed as AES128, and yet all is well. This does not make sense, but I will wait and see what happens. 

[Fabian-Out]

I opened a case as well with Meraki support and was told no backend changes were made (unannounced or otherwise) but I'm still experiencing these issues.  Luckily I was able to reference this conversation with support.  Let's see if they admit some backend changes.

[helpdeskdan]

In my org settings, I noticed a new setting for aes/des.  I'm rather certain I never set aes.  I set it back to des to match my networks.  Most, if not all, of the few random switches that had mysteriously jumped to aes now reverted back to des.  (Anticipated this, I just re-ran some python to fix my monitoring)

Tac told me:
This issue should had been resolved since yesterday.

(My answer: Ah, no....)

[RossFawcett]

Yeah, it was most definitely not resolved yesterday, and I'm hesitant to burn more time again today with it working on DES. Our response from Meraki support was a bit average, it didn't address the core issue that even when it's set to AES, it's not actually applying to everything. Have requested an update this morning to see if they can advise if it really is resolved now before I spend more time testing AES.

[GiacomoS]

@RossFawcett ,

Could you drop me a DM with the case number please? I'd like to review what happened and hopefully get some improvements in. 

For everyone else, if you are still seeing devices dropping out of monitoring and some inconsistencies on the configuration, it may just be delays in the configuration updating. If the device has not had an opportunity to fetch the config, it would continue to run on the previous until a change occurs. I'm speculating that this may potentially be the cause of more reports popping up. 

Many thanks!

Giac

[rc0123738]

Hello @GiacomoS , 

In terms of AES128 availability - should this option exist? I am still seeing it in my organization's configuration: 

All is working at the moment, but I am hesitant to change it back to DES - despite the monitoring system in use being set to DES. 

Cheers

[RossFawcett]

I've sent you a DM with the case ID, support seem to be going off on a tangent now.

Can we get some clarification on what exactly "should" work now?

At the org level we can now enable AES, now my understanding was the org level was for when querying the snmp.meraki.com? However, from what has been posted, this AES setting was incorrectly being applied to the network level?

Now the next issue, at the network level, the AES options only show up when you have a switch only network, it does not show up for combined networks. Is this expected? Or is this yet another bug?

As the network level AES setting only showing up for non-combined networks would also be contributing to what we are seeing, as really that means only two out of the 10 or so networks for this customer will ever work with AES at this stage. The rest will only be DES?

TLDR;

1. Org level AES was applying to network level
2. AES settings only show up for non-combined networks
3. Not clear whether AES is available for combined networks (does org level set this or is this DES only?)

[GiacomoS · Accepted solution]

Hey Community,

Thank you again for your patience here. 

I've managed to get to the bottom of it with the help of some internal teams. 

At this stage, only DES is supported, as per previous configuration.

AES is in development and was added into Dashboard before the intended release, which is due at some point soon. Once the feature is fully implemented and ready, you'll be able to select it directly from Dashboard for both standalone and combined networks. 

For now, please stick with DES, which is the sole mode this should be operating in at this point. 

Apologies for all the chaos that this has caused, I've run the feedback up the chain.

Giac

[rc0123738]

Thanks for the follow up! Much appreciated. 

[CarolineS]

Marking Giac's reply as the solution to this thread, for greater visibility!

[MikeHunt]

Honestly, this whole experience has not been confidence inspiring in terms of Meraki's QA & change control systems which then impacts our gear even when we've not made any changes ourselves.

I've got over $500K USD of this gear in production so far & I'm seriously considering dumbing Meraki for future sites/upgrades.

[helpdeskdan]

Is is a staggering realization when you discover other people can and, in fact, did set wrong settings on what are supposed to be "your" switches.