New firmware bringing down switches

[IT_Magician]

Last night Meraki auto pushed 14.33 firmware to switches. Switches never came back but MX online.

If we reboot MX switches come back for a bit, and then go down again.

I had a user plugin from another port on MX into switch which brings it back up. It is almost like the switch is enabling spanning tree and blocking the uplink port. Before I roll back firmware does anyone have any quick fixes for this?

[IT_Magician]

Here is the root issue:

1. MX 100 Port 3, Trunk, All VLANs to -> Port 24 on MS120

2. MS120 goes down, and when we look at MX, it shows port 3 disconnected

3. We then plugged in a new cable to port 4 on MX, and into port 20 on MS120

4. MS120 is now back online, but port 24 shows disconnected

5. Checked and no link activity on the switch

6. Enabling/Disabling port 24 which was previous uplink does not bring link back up.

[IT_Magician]

Also worth noting, after switches updated to 14.33, our MX was still on v15. We since pushed firmware to V16 but still has issue.

We then took another network with MX67w and MS120, upgraded firmware to 14.33 and had no issues. We then upgraded MX67w to v16 and still no issues. This seems like a bug between ms120 14.33 and MX100 on v15 and v16 firmware. If it happens again we are going to roll back the firmware.

[PhilipDAth]

If you have multiple links between the MX and the MS, all links bar one should get blocked by spanning tree.

If you have a single link between the MX and MS does the issue go away?

[ACaputo]

We had 14.33 apply last night and it took all of our switches offline.  Strangely rebooting out Fortigate firewall brought them back online for about 8 minutes and then they went back offline.  Rebooting the switches didn't seem to help.  I had to roll back firmware and its working fine again.  

[IT_Magician]

So just today without any firmware upgrade at all, we had a location where switch went offline. Checked the dashboard and MX is up, and shows a link light to the switch. We rebooted the MX and the switch came back online.

I am doing firmware upgrades now, just letting others know we are seeing weird switches going offline bugs still.

[IT_Magician]

For those who had issues with 14.33, it looks like they have a 14.33.1 stable release and the bug fix might knock this issue out:

Bug fixes

• Adaptive Policies rules can be incorrectly ordered resulting in traffic being blocked when it should not be

[PhilipDAth]

Are you using adaptive policy?

[IT_Magician]

No we were not.. but you know how bug fixes go, sometimes they do not cover all things that were breaking....

[sebas]

@IT_Magician Did 14.33.1 fix your issues?

[redsector]

Updateing to MS14.33 on all my networks didn´t cause this issue. We are not working with Meraki MX but with Dell Sonicwall firewalls.