Port-Channels were Defaulted during the night

DevOps_RC
Getting noticed

Port-Channels were Defaulted during the night

Last night (Thursday evening) starting at 20:30, some of our on-boarded Cisco Catalyst switches started to 'default' all Port-channels. I won't go into how impacting this was, but I'm curious if anyone else has seen this issue. I've had to remove all our switches from the dashboard as we couldn't afford to have this occur again.

12 REPLIES 12
alemabrahao
Kind of a big deal
Kind of a big deal

It does not make any sense. Are you sure that anybody changed it? Have you checked the switch log?

I am not a Cisco Meraki employee. My suggestions are based on documentation of Meraki best practices and day-to-day experience.

Please, if this post was useful, leave your kudos and mark it as solved.
cmr
Kind of a big deal
Kind of a big deal

@DevOps_RC we had the exact same last night at 1am GMT(UTC).  I was thinking we'd been hacked by a very particular nasty!  Have you logged a support ticket?  Nothing else changed but all six port channels had their config defaulted on our pair of 9300s.

MerakiLuke
New here

Hi All,

 

Just toss my hat into the ring we had the same thing happen to our Catalyst 9300 about 1:00am GMT.

 

 

DevOps_RC
Getting noticed

I confirmed before posting this that no-one had logged onto the switches, or even had an open session and made any changes. But it's fair to ask the question. I've raised a support ticket with Meraki and thankfully I also have good relationship with the sales/technical reps at Cisco and Cisco Meraki..Can I just say Cisco and it covers both?? Anyway, it's looking likely that there was a change recently in relation to Catalyst monitoring in Meraki, but not had confirmation yet that it was absolutely the cause of the issue. I'll update once I have further details.

ConnorL
Meraki Employee
Meraki Employee

Hi @cmr  @DevOps_RC  @MerakiLuke @alemabrahao ,

 

Please follow this thread for the latest information about this issue: https://community.meraki.com/t5/Meraki-Service-Notices/Cloud-Monitored-Catalyst-switches-issue/ba-p/...

 

Kind regards,

 

Connor.

Thanks for the update. Hopefully this can get rectified quickly, and prevented from happening again, so I can on-board our switches again. 

cmr
Kind of a big deal
Kind of a big deal

@MerakiLuke just changed the Meraki user to privilege 1 for us...

Rich-D
Here to help

Same for me on my core C9500 stack.  Ugh!!  Definitely thought it was malicious.  

cmr
Kind of a big deal
Kind of a big deal

@DevOps_RC and @Rich-D have Cisco reached out to you with next steps and did you get a banner?  We haven't had either...

DevOps_RC
Getting noticed

@cmr No banner on the dashboard. No update on the support call I raised since the last one yesterday afternoon, however I have had emails from a few Cisco reps. I've been advised, that the change that took place on Thursday night has been rolled-back and Cisco have a detailed understanding of what happened and discussions are already under way to define what needs to be put in place to prevent this from happening again.

Until then all our switches have been removed from the dashboard.

With regards to changing the user privilege level, that would be a temporary fix, however I suppose you would need to make sure that the relevant commands are added to that privilege level for the dashboard to still have access to the information it gathers including 'show run'. Maybe someone from Meraki could advise which commands it needs to be able to run?? Please.

Rich-D
Here to help

I did get the banner on my dashboard page.  According to this it is resolved now. https://community.meraki.com/t5/Meraki-Service-Notices/RESOLVED-Meraki-Dashboard-monitored-Catalyst-...

 

I did in the short term since I'm going to be out of the office change my "meraki" user on the cat switches to privilege 1 (read only) from 15.  

DevOps_RC
Getting noticed

After receiving an update from Cisco (I am unable to share any information on this), I have re-on-boarded our Catalyst switches into the dashboard. I think there have been some updates to the on-boarding, as all of my stacks on-boarded first time with all members of the stacks. Previously on the odd occasion, some members wouldn't show in the dashboard. Also, clients connected on non-managed vlans and port-channel interfaces are also showing correctly, previously it seemed to only show clients on the switch on-boarded vlan interface. So Kudos to Meraki to continue to develop both on-boarding and monitoring of Catalyst switches. If Meraki are taking requests, could you please add a 'recommended IOS version' for each of the on-boarded switches to alert us of a suggested upgrade for them, and ideally add the functionality to perform the upgrade to the IOS similar to the current options for Meraki switches, i.e. Now or Schedule it.

Get notified when there are additional replies to this discussion.