[Switching Public Beta] Be the First to Experience the Cloud-native IOS XE on Meraki Dashboard!

[Minyi]

Introducing the cloud-native IOS XE on the Meraki dashboard! With this release, we are beginning an exciting transition from a container-based architecture to one based on native IOS XE with support for Meraki cloud management.  

This drastically simpler architecture unlocks many benefits for your cloud-managed Cisco Catalyst switches, including the C9300-M, C9300L-M, C9300X-M, and MS390 families. Some benefits include faster boot and initialization performance, especially for stacks and the start of a new generation of capabilities we will deliver with faster speed. It also introduces the ability to perform CLI show commands directly from Dashboard! 

Please refer to this announcement post for more details. 

Share Your Feedback

We value your feedback on our latest release! Please take a moment to complete this brief 5-minute survey (https://forms.office.com/r/gHuK4bJRZ5) and share your experience with us. 

[PhilipDAth]

I am keen to try this out.

Is the new naming convention, IOS-XE, mean the end of the CS naming convention far into the future?

[Minyi]

Hi @PhilipDAth, how are you doing? That's correct, CS17 will be the last CS-branded firmware release, and we will be transitioning to cloud-native IOS XE moving forward. There will be a time for these two firmwares (CS and cloud-native IOS XE) to co-exist on the dashboard until the natural progression. 

[PhilipDAth]

Are we expecting to lose any functionality in this change to native IOS-XE, for this release?  Or do we think all existing features will be operational?

[Minyi]

Great question - please review the temporary feature gaps on this overview page:  https://documentation.meraki.com/MS/Cloud-Native_IOS_XE/Cloud-Native_IOS_XE_Overview

[cmr]

When can we expect the DHCP server to work properly? i.e. with default router and domain name options possible.

[WirelesslyWired]

We should have the DHCP server configuration resolved soon. I will check with engineering and get back to you. 

[WirelesslyWired]

Ok just checked and the fixes should be in production for the beta. If you have a chance to try it @cmr , please do and feel free to reach out to me alburger@cisco.com if you find issues. From testing we are seeing things working the way it should, but would be glad to have some field feedback. 

[cmr]

Hi @WirelesslyWired thank you for the reply.  I have just tried setting the default gateway and domain name options in a DHCP server on a C9300L and I get the the below:

Hopefully it will make the next update?

[PhilipDAth]

My initial experience, I crashed and burned with this upgrade.

I have test trunk ports setup both with and without SGT.  It is the easiest way to restore access to a switch if you screw the SGT config up.  I normally use the SGT trunk ports.

After the switch had been offline for an hour following the upgrade, I switched across to using the trunk port without SGT.  It still required a power cycle, but then came online.  I then changed back to using the trunk ports that had SGT enabled, and everything continued to work.

My suggestion - don't attempt to do the upgrade if you have SGT enabled on your trunk towards the Internet.

Tools/Reboot Device no longer works.

The Dashboard Throughput tool does not work.

The terminal function doesn't work.  It reports an error no matter what command you try.  Even ? for help returns an "internal error".

Blink Leds does not work.

I couldn't retrieve the MAC forwarding table.

[MRCUR]

The tools (including CLI) work for me on a migrated MS390. It seems like your migration might not be done yet or something else is going on. 

[PhilipDAth]

I'm testing on a C9300.

[cmr]

On a C9300L with no Adaptive Policy I get different results to you:

Reboot switch - works

Dashboard Throughput - fail

Terminal - works, but only if everything is lower case, i.e. show and not Show

Blink LEDs - works

MAC forwarding table - works

[PhilipDAth]

Have you experienced issues with it randomly crashing (this is what prompted me to roll back)?

[cmr]

Not yet, though the longest uptime I've had was about 4hr30 as I tested the reboot switch command!  It's in my house hallway and my wife doesn't like "that awful noise", so I only have it on in the day when testing... 😬  Once I've completed more tasks I'll move it somewhere out of the way so I can leave it on for a few days. 😎

[PhilipDAth]

This is a bit confusing as well.  Does that mean the config on the switch is up to date, or it hasn't applied a change made 27 minutes ago?

[MRCUR]

I think this should say "last fetched" once on IOS-XE. 

[cmr]

On the new version of the switch page it now does!

[PradeepK]

This is such a cool feature!! Customers would love it

[PhilipDAth]

I managed to get 12 hours of run time out of the switch and then it crashed.  I have got support taking a look now.

[cmr]

So, better that a 390 then 😈

[PhilipDAth]

I'm giving up on this code train.  I only managed to keep it running 1 hour before crashing this time.

This code train is VERY unstable.

[cmr]

Ouch!  Out of interest, what model(s) of 9300 did you try and were you using advanced features, or just the usual L2/L3 switching and routing?

[PhilipDAth]

My unit is a C9300-24UX.

I mostly had SGT enabled.

[cmr]

@PhilipDAth I'm guessing this is after your experience, but this was added to the release notes:

Known issues

• Adaptive Policy and SGT configurations are not currently supported

[PhilipDAth]

Haha.  That is a good outcome.

I'm working with support at the moment to try and identify the issue further.

[espenvp]

Had a similar experience with a c9300x-24 stack. Firmware upgrade went smooth for this stack along with 20'ish other c9300's however I had one stack "die" aprox one hour after IOS_XE boot for the first time. See timeline in the attached image. Is support/RMA the only solution to this? 🙂 

[cmr]

@espenvp what features were being used on that stack?  i.e. adaptive policy etc.

[espenvp]

Nothing fancy...

• named vlans / vlan groups
• vlan profiles
  • mostly to counter the default "allowed vlan 1-1000 range"
• multi chassis etherchannel using LACP/Aggr uplink ports

That's pretty much it, very default as its just a distribution switch. No AGP just make that clear 😉 

[cmr]

Did your port names survive the upgrade?  I have a 9300L where the port configs remained, but the names were all lost...

[espenvp]

No, all ports had their names overwritten with the default catalyst naming scheme "GigabitEthernet1/0/xyz, TwentyFiveGigE1/0/xyz" etc. My network is currently running C9300L-24P and C9300X-24Y models.

[cmr]

That's exactly what I had, time to add it to the known issues list...

[cmr]

@espenvp are you using the old version of the switch page, try the new version, for me the port names that were there pre IOS-XE all return.  The power section also seems more correct (no phantom power stacking in my case) however the packet counters only seem to work with live data...

[espenvp]

Yes, I'm using the "old" switch page and the names return using new switch page (thanks for tip), however i find the new page a bit lacking currently such as the stacking indicators don't work and the power tab is missing. The LLDP/CDP column in the ports page doesn't work either. So I'm going to stay in the "old switch page" for now 😉

[cmr]

I agree that the new switch page needs some work, however you should find the power information under the device health tab 🙂

[espenvp]

hahaha found it 🤣

[PhilipDAth]

Did it work again after you power cycled it?

[espenvp]

No, it seems bricked, stuck in rommon or thereabouts, to bad the CLI port is disabled. The local management page is available but not able to log on using "known" credentials (factory default and dashboard ones). Factory reset has been performed with no result. 

[cmr]

How much do you get on the console port? 

With the C9300L I have, the console port is live for about 3 minutes at the start of the boot and does display bootloader version, 802.3bt support, stack unit number and all the MAC/serial number data. 

[espenvp]

It seems to me that the console port stops working after the local management page becomes available.

[cmr]

That would make good sense!

[Brennan_Martin]

Hey @espenvp - have you already opened a support case to have us take a look at this? If the local status page is available, the switch isn't bricked, and likely that a level 3 reset at worst will restore the device to factory settings, but please call in and let our team have a look. 

the local status page credentials were also recently changed, but support can help you with that as well.

[cmr]

@Brennan_Martin "the local status page credentials were also recently changed".  Do you have a link to the documentation for this?

[Brennan_Martin]

You bet. 

MX devices running MX 19+ firmware, MS devices running MS 17+ firmware and MS390/Catalyst devices running CS 17+ firmware will use the username admin and the password will be the serial number of the device (upper case letters and dashes). 

https://documentation.meraki.com/General_Administration/Tools_and_Troubleshooting/Using_the_Cisco_Me...

[espenvp]

I was told by support that the local status page needs to be accessed using capital A in Admin as username and then the serial as password. Also case sensitive and normally is in all caps.
Meraki documentation says otherwise. 

[espenvp]

[PhilipDAth]

I couldn't get my admin page to work, and a level 3 reset did nothing in my case (on the console it reports it is doing a level 3 reset - but it doesn't actually do anything).  @espenvp I am interested in your case - as it sounds like the same thing has happened to your C9300 as happened to mine.

[espenvp]

With a level 3 reset you mean "hold mode button for more than 21 sec" ?

Under "Factory Reset Button"

https://documentation.meraki.com/MS/MS_Installation_Guides/Catalyst_9300-M_Series_Installation_Guide

[PhilipDAth]

For my bricked C9300, the Ethernet ports can not get a link light, and the stack ports no longer work.

Is yours the same - have the Ethernet ports completely stopped working?

[espenvp]

Yes my switch is totally offline, there is no link seen from the upstream core switch.

[PhilipDAth]

@espenvp that pretty much confirms it - both of our C9300s have exactly the same symptoms.

Meraki are working hard on this one so I am sure they are already all over it, but it you want you could mention my support case #12337148 in your support case, so they both get linked back to the same internal investigation.

[PhilipDAth]

Things got worse.  I downgraded from IOS XE 17.15.1.9 to CS17.1.3.1, and it left the C9300 dead.  None of the Ethernet ports work anymore.  No link lights.

Support is now arranging an RMA for me.

[cmr]

Sounds like MR31.1.5, but for switches 😬

[cmr]

To be fair the C9300L I am running the IOS-XE beta on does seem to be stable, however some of the traffic reporting seems a bit odd and the bars for whether a port is up or down and what speed it is connected at are out of whack:

The pink arrow points to where the switch was powered on, the blue bar is for "1Gbps" connectivity and the green bar is for "10Mbps"...  I think the blue bar is actually denoting 10Mbps as the connected PC was in standby and the NIC does throttle down, however I'm not sure why it isn't grey before 7am when the switch was off...

[Brennan_Martin]

Hey Philip, thanks (as always) for the feedback - sorry you’re having a poor experience. Generally this beta is working pretty well, but it sounds like this switch is likely stuck in ROMMON. I’d like to make sure we’re properly investigating - would you mind sending me the case number via DM and I’ll have the team take another look?

Will we see you next month at CL MEL? Would love to sit down and share a bit about what we’re doing and get your thoughts.

[PhilipDAth]

Done, and I plan to be at Cisco Live Melbourne.  🙂

[MRCUR]

Thankfully I haven't had the issues like @PhilipDAth with a migrated MS390, but I do have a C9200L that seems to be happy never kicking off the migration. Definitely not enough info through CLI or Dashboard to tell what's going on or what's not going on. 

[WirelesslyWired]

@MRCUR On the 9200L, you have claimed the switch into a network but it has not migrated correct? 

[MRCUR]

@WirelesslyWired Yes that's correct. 

[WirelesslyWired]

Did you open a ticket by chance? If not, please do and forward me the ticket ID at alburger@cisco.com

[NetEngJH]

Nice to see a sense of humour still exists at Cisco! Worth upgrading just to press this button.

[GIdenJoe]

Niiice 😉

[MRCUR]

Has anyone tried RADIUS with a C9300 that's been migrated to native IOS-XE? The switches keep logging "sessmgrd: Cannot open UDP socket for service Radius server" and we don't see any traffic to the configured RADIUS servers but we do see it in the aaa config. 

[espenvp]

To me it seem like a connection problem between the management IP of the switch to the radius server. Is the management IP of the switch defined as a network device and authorized as a radius source on the raduis server? Is the appropriate firewall and routing in place between the switch and radius?

I currently have a working dot1.x/mab deployment using radius (ISE) with IOS-XE on the switches, seems to be working as intended 🙂

[MRCUR]

Found the issue: IOS-XE mode switches do not include the Message-Authenticator attribute even though in CS mode they do (MS switches also include it). 

[DevOps_RC]

For info, I've been notified of a bug with long allowed Vlan statements which IOS breaks into multiple lines. So if you have an allowed vlan over ~39 characters (including ,-) it will fail configuration validation. I hope this is fixed soon, so I can get IOS-XE onto the last few of our MS390s..I feel  IOS-XE beta is a more stable/robust firmware than the MS/CW fudges to get the MS390s (C9300 HW) into the dashboard. 

[DevOps_RC]

Long allowed Vlan statements (over ~39 characters) has now been fixed, but the maximum character limit is still ~240. Please also note if your device is not upgrading and stuck at saying 'Not running configured version', make sure that the 'Local device status pages' (Under Network>General Device Configuration) is enabled. I was advised on this by Meraki, good spot to the engineer who figured that one out.