Last night (Thursday evening) starting at 20:30, some of our on-boarded Cisco Catalyst switches started to 'default' all Port-channels. I won't go into how impacting this was, but I'm curious if anyone else has seen this issue. I've had to remove all our switches from the dashboard as we couldn't afford to have this occur again.
@DevOps_RC we had the exact same last night at 1am GMT(UTC). I was thinking we'd been hacked by a very particular nasty! Have you logged a support ticket? Nothing else changed but all six port channels had their config defaulted on our pair of 9300s.
I confirmed before posting this that no-one had logged onto the switches, or even had an open session and made any changes. But it's fair to ask the question. I've raised a support ticket with Meraki and thankfully I also have good relationship with the sales/technical reps at Cisco and Cisco Meraki..Can I just say Cisco and it covers both?? Anyway, it's looking likely that there was a change recently in relation to Catalyst monitoring in Meraki, but not had confirmation yet that it was absolutely the cause of the issue. I'll update once I have further details.
Please follow this thread for the latest information about this issue: https://community.meraki.com/t5/Meraki-Service-Notices/Cloud-Monitored-Catalyst-switches-issue/ba-p/...
Thanks for the update. Hopefully this can get rectified quickly, and prevented from happening again, so I can on-board our switches again.
@cmr No banner on the dashboard. No update on the support call I raised since the last one yesterday afternoon, however I have had emails from a few Cisco reps. I've been advised, that the change that took place on Thursday night has been rolled-back and Cisco have a detailed understanding of what happened and discussions are already under way to define what needs to be put in place to prevent this from happening again.
Until then all our switches have been removed from the dashboard.
With regards to changing the user privilege level, that would be a temporary fix, however I suppose you would need to make sure that the relevant commands are added to that privilege level for the dashboard to still have access to the information it gathers including 'show run'. Maybe someone from Meraki could advise which commands it needs to be able to run?? Please.
I did get the banner on my dashboard page. According to this it is resolved now. https://community.meraki.com/t5/Meraki-Service-Notices/RESOLVED-Meraki-Dashboard-monitored-Catalyst-...
I did in the short term since I'm going to be out of the office change my "meraki" user on the cat switches to privilege 1 (read only) from 15.
After receiving an update from Cisco (I am unable to share any information on this), I have re-on-boarded our Catalyst switches into the dashboard. I think there have been some updates to the on-boarding, as all of my stacks on-boarded first time with all members of the stacks. Previously on the odd occasion, some members wouldn't show in the dashboard. Also, clients connected on non-managed vlans and port-channel interfaces are also showing correctly, previously it seemed to only show clients on the switch on-boarded vlan interface. So Kudos to Meraki to continue to develop both on-boarding and monitoring of Catalyst switches. If Meraki are taking requests, could you please add a 'recommended IOS version' for each of the on-boarded switches to alert us of a suggested upgrade for them, and ideally add the functionality to perform the upgrade to the IOS similar to the current options for Meraki switches, i.e. Now or Schedule it.