Hello,
At a remote location, we have 4 switches(MS350) running in a stack and performed a routine upgrade from 10.25 to 11.22. After the upgrade we started seeing the following:
Meraki performed packet captures and said they did not see return traffic from the upstream devices. Initially Meraki thought the issue was 11.22 (as there is a known bug related to stack switches in the release). They recommended the following:
None of those next steps fixed issues 1-6. The network performance got even worse, we lost ospf connectivity between the switch stack and our mpls router. Per Meraki, the switch stack was having issues fetching its configuration, ospf was no longer established and now the site was down hard. The switch stack in the portal alerting something similar to "stack configuration in member different from dashboard"
After finally getting to the Meraki escalation team (going through 5-6 engineers, 2 days later, escalating to account team and finally a down hard network), they were able to isolate the issue to the following: port mirroring on the stack triggered an issue with switch stack being unable to fetch its configuration from the cloud. We removed ALL port mirroring, rebooted the switch stack, ospf re-established and issues 1-6 immediately cleared. The site is back up and performing normal.
UPDATE 12/14: We continued to have poor performance (demonstrated by high packet loss, error messages indicating that the merakis were having a difficulty reaching the cloud, user complaints of slowness, etc) with the switch stack (after all the changes above). We ended up having to remove the stack....no reported issues after removing the stack.
Oh noooo, thank you for sharing.
Did you restore the port mirroring after you were done, or leave it off?
Hello Nash, we have not enabled port mirroring. We are going to wait a few days and re-enable.
@sdh1972 Owch what a painful excercise, thanks for sharing though. Were you actually using the port mirroring or had it been setup in the past for something and forgotten?
I had problems with 11.22 also, see my thread "11.22 killed my call center".
I rolled back to 10.45 though instead of going forward to 11.28. It sounds like I need to stay on 10.45 if I can't use port mirroring on 11.28. I have mirror ports going to a security device, looking for lateral movement of malware. I have no intention of disabling this.
@Brons2 wrote:I had problems with 11.22 also, see my thread "11.22 killed my call center".
I rolled back to 10.45 though instead of going forward to 11.28. It sounds like I need to stay on 10.45 if I can't use port mirroring on 11.28. I have mirror ports going to a security device, looking for lateral movement of malware. I have no intention of disabling this.
That is a safe way to go for sure, so I respect that.
That being said, I am curious though if it was only because port-mirroring was enabling 'during' the upgrade process that it broke stuff.
I wonder if you turn it off 100%, perform the upgrade, and then re-enable post upgrade if it would still cause issues or not. Obviously I don't expect you to find out with your production environment of course lol, but just food for thought.