C9300L losing IP config

[aironfabio]

Hello team,

I have a tenant with both MS130 and C9300 with Meraki firmware switches.

The C9300L-48P-4X stack (2 appliances) periodically loses network connectivity.

All switches are layer 2 only; the stack keeps switching packets, downstream devices (such as one MS130) work just fine and can be managed by the dashboard. Yet the C9300 stack doesn't respond to ping and of course can't communicate with the dashboard.

a hard reboot solves the issue, then after a week or so it loses once again IP connectivity. It happened three weeks in a row, with no discernible pattern for time.

These 2 are the only Catalyst switches in the tenant; all the MS130 and all Access Points in the tenants show no issues. All devices have static IPs on the same subnet.

can you please suggest a few troubleshooting steps for this issue?

thanks in advance

Fabio

[Mloraditch]

Are you on CS Firmware or IOS-XE? If on CS, I suggest trying IOS-XE or going to the latest CS if not on it. It is known that the Container that runs on the CS firmware sometimes just crashes and a reboot is required.

If you are already on IOS-XE, I'm presuming you mean you can't ping even locally or see the device online in any state and then the following might help.

Are you using a Static IP or DHCP? If DHCP, have you done a packet capture to see if what is happening when the connectivity is initially lost? Secondarily have you tried switching to a static IP to see if that has the same problem? If your switch is already static, this is a stranger problem and I'd be doing packet captures on the port it uplinks to see if you can see any of the management packets and what's happening with them."

Have you tried accessing the local status page? Does it say anything?

[aironfabio]

thanks for the reply!

IP is static. I am on the latest CS firmware, and the fact that "it is known to crash" is unfortunately news to me, I usually operate with Meraki-native appliances.

unfortunately I am not onsite where the appliances are deployed and I have no other switches closeby to configure an out-of-band vlan for local status page... I keep asking the maintenance operator to reboot the appliances.

is there a known source that testifies that CS firmware crashes a lot and we should switch to IOS-XE? The continuous reboots are unsustainable.

I will set up a packet capture on the uplink, in the meantime any other input is appreciated.

[Mloraditch]

There is not a posted known issue, in the latest CS release notes. They actually fixed a bug in 17.2.3 related to the container. Regardless, It's just been something that has been experienced often by users and that problem will no longer exist once on Native IOS-XE

[ElwinNetworking]

Seen this happen a few times on CS firmware as well.

The switch keeps forwarding traffic but the management plane just stops replying, looks like the container crash Mloraditch mentioned. I’ve also seen it come back online after a reboot for a few days before it dies again, no clear pattern either.

If you can grab logs right before the reboot you might catch the process restart event Switching to native IOS XE seems to be the only reliable fix so far, haven’t seen the issue again after that.