IOS XE 17.15.2 not saving config changes

[Bovie2K]

We have two Cisco 9300L's in a stack that we had all configured and were working fine. Then we put them into production and now when we update port configuration it's not actually updating on the switch. VLAN's are staying the same. We can confirm this  by going into the terminal and doing a show running-config interface te1/0/39 for port 39 and it shows the old config on the port. They were updating fine before we did the move. The VLAN config is the main thing but we also noticed if were turn off POE and disable the port completely it doesn't update the config the port says working. 

The other problem we are having and we are trying to track down is every time we make a config change the stack is doing an RSTP update and dropping all traffic for 10 to 15 seconds which I'm not sure is related or not. We can see the RSTP updates in the logs.

We went with IOS XE 17.15.2 because its the future and also the CS firmware had broken 802.1x in it. Turns out it's also broken for us in IOS XE but we haven't been able to solve with support yet. the 802.1x config works fine with the MS firmware for the Meraki style switches.

[Mloraditch]

It seems you are already working with support and outside of doing a factory reset, that's going to be your best bet to resolution.  I'm not aware of the 802.1x bug you are referencing in the older CS code and there is no known issue listed, but Meraki has not been historically published all the active bugs. If you haven't already the last thing you can try with supports help is reverting to the CS firmware. I do have 9300Ls active with 802.1x running on them and no issues. I'm using EAP-TLS and MAB with ISE. 

[Bovie2K]

Thanks for the info. We're using 802.1x with NPS and just can't get it to work. Still diagnosing with support.

As for the config issue unfortunately support doesn't have anyone that knows IOS-XE that works over the weekend. We were told to call back Monday. So the downgrade is out of the options as well. 

Do you know if the factory reset is the same as it is with the MS? It will wipe it then connect to the cloud and redownload the new config?

[Bovie2K]

Also seeing this error now in the error log 

CRIMSON
3
DATABASE_MEMLEAK
Database memory leak detected in /tmp/rp/tdldb/0/IOS_EVENTQ_DB database, queue_syslog_events_msg_record/syslog_events/queue_record size:120 byte, callsite:

[Bovie2K]

This could have started when I used sticky Mac. I see it's not supported by IOS-XE even though I've removed the sticky Mac config I still have the same problem. not sure why the GUI lets you configure sticky Mac.

[cmr]

@Bovie2K I'd suggest this is the case.  @PhilipDAth had problems with beta 1 and a feature that was then declared to be unsupported.

Using a beta in production is usually not advised, though I can see the temptation, I did think about the same for a C9300-24UX stack, as my test 9300L has been running it well for a while, but decided it was too risky.

As this is a beta of a new management model, I would advise regularly reading the release notes as features are changing from supported to unsupported and vice versa due to results out in the field.

In @PhilipDAth's case, I believe the downgrade led to the switch needing to be RMAed, so I'd work carefully with support on this as you may need to do the same.

Good luck!

[Bovie2K]

@cmr ya I get it. Normally I always run stable but like I said I was trying to diagnose an 802.1x issue and knowing this is where Meraki was going I figured it better to make the switch and see if it worked. The firmware seemed stable so we made the swap. I did read the release notes but of course didnt remember the sticky Mac if that is the cause I would agree it's too risky to leave something to memory where the GUI doesn't stop you. I'll call support in the morning and see if they can fix and will report back so we all can learn.

[Bovie2K]

Ok I think we know what is going on. When you make a config update the switch goes through some back end tests before downloading the config. One of those tests is a RSTP / MSTP test and thats failing. That explains why we are seeing an RSTP reload in the logs and it's not downloading the config. Support thinks we are effected by a bug and they are looking into it.

[JeroenVercoulen]

I'm having the same issue here. With 2 Meraki C9300 stacks. Also a Factory reset isn't working just to let you know. I've tried about everything. Even bringing 1 switch home to make sure no firewalling what so ever is present.

[Bovie2K]

Meraki just fixed for me this morning. I'm waiting for the root cause but they said they did something on the back end and now it works.

[cmr]

Excellent, +1 beta testing star ⭐ 

[JeroenVercoulen]

They did something on the backend for me also, but that didn't fix the switches I factory defaulted. The still running stack they didn't fix yet.

[JeroenVercoulen]

Comment from Cisco

Our engineering teams pushed a fix for this issue late last night. Please test and let me know if you are now able to make successful configuration changes, from my perspective everything appears to be working correctly at this time.