ALERT: Cisco 9000 switch port channel config being erased in monitor mode

cmr
Kind of a big deal
Kind of a big deal

ALERT: Cisco 9000 switch port channel config being erased in monitor mode

Please be aware that we and at least one other company have had all port channel configs defaulted on Cisco Catalyst 9000 switches that are connected in monitor mode to the Meraki dashboard.  See discussion here: Re: Port-Channels were Defaulted during the night - The Meraki Community

 

In our case it was a stacked pair of 9300s last night at 1am UTC and all six port channels had their configuration defaulted at the same time.  Nothing else changed at all.

 

I have logged a support ticket asking for urgent developer involvement.

7 REPLIES 7
Brash
Head in the Cloud

Jeez that's messy! Interesting to know what caused it for everyone to be impacted at the same time.

I had expected something like that might happen in management mode but the fact it occured in monitor mode is deeply concerning.

ConnorL
Meraki Employee

Hi @cmr @Brash ,

 

Please follow this thread for the latest information about this issue: https://community.meraki.com/t5/Meraki-Service-Notices/Cloud-Monitored-Catalyst-switches-issue/ba-p/...

 

Kind regards,

 

Connor.

cmr
Kind of a big deal
Kind of a big deal

@ConnorL why after 8 hours of being acknowledged is there no update?  This took down one of our venues for several hours while we tried to figure out what had gone wrong, not what we expect from Cisco.

Brash
Head in the Cloud

The thread has been updated to outline that engineering identified the impacted devices and Meraki is reaching out to customers.

 

What it's lacking however is any detail around why it occured, and whether further safeguards are being put in place to prevent anything like this from occurring in future.

I would expect monitor mode to be completely hands-off in terms of the ability to change configuration on a switch.

cmr
Kind of a big deal
Kind of a big deal

We didn't get a banner, but had already reported, so no need I guess.  However as we set the Meraki user privilege to 2 we now have a message saying that the switch config is out of date, so that at least means something has been tried.

cmr
Kind of a big deal
Kind of a big deal

Or perhaps the dashboard tried to delete the port channels again... 😱

cmr
Kind of a big deal
Kind of a big deal

Our ticket was also not updated so I've chased, as we're no further forward in knowing what the next steps are supposed to be 😡

Get notified when there are additional replies to this discussion.
Welcome to the Meraki Community!
To start contributing, simply sign in with your Cisco account. If you don't yet have a Cisco account, you can sign up.
Labels