MS350 Stacking Issues

[bholmes12]

I have 3 MS350 stacks, 6 switches in each stack, running on 8.10 software. 

All stacks have been up and running without issue for the past month or so. No changes have been made on the Meraki switches or the Cisco Nexus 9k's that are upstream. 

This morning within an hour all 3 stacks had switches that removed themselves from the stack and stopped forwarding traffic. 

Stack 1 - 1 switch disappeared from the stack and stopped forwarding traffic - Did not come online until after a reboot. Uplink port was not on this switch. 

Stack 2 - 1 switch disappeared from the stack and stopped forwarding traffic - Did not come online until after a reboot. Uplink port was not on this switch. 

Stack 3 - 5 of the switches disappeared from the stack - 1 of the uplinks was on one of the switches that disappeared from the stack. After rebooting the switch with the uplink the 5 switches rejoined the stack. 1 switch in this stack stayed online, that switch has an uplink to the core. 

It seems really strange that 3 stacks had the same issue within an hour, Meraki support suggested we hit a bug (8.10) and have requested I upgrade to 9.19. 

Has anyone else been having stacking issues? 

I typically put in Cisco 4500's or 3850's and these are my first 3 MS350 stack closets. This obviously isn't giving me a very warm and fuzzy feeling to have a failure like this after a month or so. 

Curious what other users experiences  have been when using MS350 stacks ???

[BHC_RESORTS]

---

@bholmes12 wrote:

I have 3 MS350 stacks, 6 switches in each stack, running on 8.10 software. 

All stacks have been up and running without issue for the past month or so. No changes have been made on the Meraki switches or the Cisco Nexus 9k's that are upstream. 

This morning within an hour all 3 stacks had switches that removed themselves from the stack and stopped forwarding traffic. 

Stack 1 - 1 switch disappeared from the stack and stopped forwarding traffic - Did not come online until after a reboot. Uplink port was not on this switch. 

Stack 2 - 1 switch disappeared from the stack and stopped forwarding traffic - Did not come online until after a reboot. Uplink port was not on this switch. 

Stack 3 - 5 of the switches disappeared from the stack - 1 of the uplinks was on one of the switches that disappeared from the stack. After rebooting the switch with the uplink the 5 switches rejoined the stack. 1 switch in this stack stayed online, that switch has an uplink to the core. 

It seems really strange that 3 stacks had the same issue within an hour, Meraki support suggested we hit a bug (8.10) and have requested I upgrade to 9.19. 

Has anyone else been having stacking issues? 

I typically put in Cisco 4500's or 3850's and these are my first 3 MS350 stack closets. This obviously isn't giving me a very warm and fuzzy feeling to have a failure like this after a month or so. 

Curious what other users experiences  have been when using MS350 stacks ???

---

The MS line is the one product we don't run in our environments, so my advice is more theoretical than practical I'm afraid.

Did the logs show anything interesting regarding the stacking? Also to be clear, we are talking about physical stacking here, not virtual stacking, correct?

Normally I would blame the upstream Nexus 9ks, as those have more bugs than a Brazilian jungle, but since some of the stacks don't uplink to it, i'm not so sure.

[bholmes12]

@BHC_RESORTSYeah talking about physical stacking. 

Nothing in the logs, just all of a sudden certain switches were no longer seen as in the stacks. 

We probably have 30+ stacks hanging off the 9k's the MS350's were the only ones that had any issues.

It did get me wondering if Meraki pushes out patches to switches outside of just standard firmware upgrades??????

[BHC_RESORTS]

Strange. Well, the beta firmware is 9.26, and while it isn't always a great idea to run beta in a production environment...the stable track is pretty far behind at this point, so I'd weigh your options. I skimmed through the release notes, and I didn't see this specific issue mentioned, but there are a TON of stability and bug fixes, so you never know. We run two MS2208p, and are on 9.26 with no issues yet. Small sample size, I know.

[NFL0NR]

we have 2 425's stacked... I know not the same switches but if there's any questions about how ours are setup I'd be happy to check our setup... 

we haven't run into any issues once we got it setup correctly.

[jfry2k]

Make sure Meraki support has the "secret new firmware on them."  We implemented some of these a few weeks ago as well and had to call support to get the firmware.  FYI your switches will show "up to date" without it.  

[Dave]

I have 2 MS-350 stacks.  One stack with 2 switches and the other with 3 switches.  

When we first tried deploying these a year ago we had issues.  Firmware wasn't out for stacking them back then.  We had multiple engineers pulling their hair out of some of the bizarre issues that were going on. 

I haven't had any issues over the last 8 months with them though.   We are still running MS 8.10

[bholmes12]

@Dave@NFL0NR @BHC_RESORTS

Ok so today again I had the some of the switches fall out of the stacks, but I think we have been able to track down the issue. 

Our security team was running ping sweeps across our network. When the sweep got to the MS stack closets for some reason it took down the OSPF neighbor relationship from the MS 350's to the 9k's. 

When the OSPF neighbors went down and then came up the MS switches hit a bug. Based on what Meraki support is saying there is a bug when the uplink goes down other switches in the stack fall over the stack. 

Tonight I am upgrading to 9.19 and Meraki support believes the stacking bugs are fixed in this release. 

[Dave]

Hope the update goes smoothly for you and resolves the issue. 

[NFL0NR]

let's hope newer is better...

[Mr_IT_Guy]

---

When the OSPF neighbors went down and then came up the MS switches hit a bug. Based on what Meraki support is saying there is a bug when the uplink goes down other switches in the stack fall over the stack. 

---

@bholmes12 Did support say that this was only on the MS350's or does it affect all stacked switches??

[bholmes12]

I believe it's all switches on 8.10. 

But it sounded like it only effects switches in the stack that don't have there own uplink. So in my case, I have 3 stacks of 6 switches, each stack has 2 switches with uplinks. The other 4 switches in the stacks don't have there own uplink, it only seems to affect the switches without uplinks. 

They also mentioned that in general there are a lot of stacking bugs on 8.10. 

One of the bugs is actually upgrading the switches! So instead of just scheduling the upgrade they have a very specific process they want me to follow for this upgrade. 

1. Power down all but the primary uplink switch

2. Run the upgrade on the one powered on switch

3. boot up the remaining switches, I'm not sure if they meant all at once or one at a time. 

But they mentioned they have had lots of issues when upgrading off 8.10 with stacked switches. 

[Mr_IT_Guy]

Please please please!!! Let me know how this goes. May need to discuss this with my team in the morning.

[bholmes12]

Upgrade is completed, ended up going to beta 9.26 instead of candidate release 9.19. 

The tech I talked to tonight said that if you are below 9.2 and going to 9.19 then have had stacked switches hang, but that issue was fixed in 9.25, 9.26 has since been released to he advised to go to that to avoid any of the issues of the switches locking up. 

When I started the upgrade all switches started flashing white on the front while they were downloading the image. After about 35 minutes of that they rebooted, the reboot was quick I would say the stacks were back up and passing traffic in about 3 minutes. 

I hadn't seen this before, but I was able to watch the upgrade process until reboot on the http status page. I got a little nervous when for about 20-25 minutes all of the switches were hung at 66%. But eventually they jumped to 100% and rebooted. 

I guess the real test will be over the next few days. 

[Dave]

Thanks for the update and all the detail.   I'd have been nervous too during that long upgrade process. 

This is something I'm going to be doing in the near future.  We have random dropped uplinks from time to time.   Let us if it fixes your issues. 

[BHC_RESORTS]

Firmware updates are the only thing that somewhat scares me on Meraki products. Not having your trusty serial connection to xmodem over an update in case it didn't work...99% of the time, they go smoothly, but when they don't...

[MRCUR]

Stacks dropping their uplink (and then doing absurd things afterwards) is a well known issue on the MS team from what I've gathered. The uplink being dropped is specifically called out as a bug fix beginning with firmware version 9.27 which was just released. 

[MRCUR]

As a followup from my original post mentioning 9.27 - I now have three buildings running 9.27 and it's been fantastic. Easily the best MS firmware I've used so far. Stacks are actually behaving correctly for the first time ever, including stacks with uplinks spread across switches. 

[Dave]

Excellent.  That's good to hear.  Thanks for the update. 

[Mr_IT_Guy]

Thanks for the update!! Might have to talk with the boss on this one.

[bholmes12]

I have been on 9.26 for a few weeks now, I have had one issue. 

The building had a power outage taking my 3 MS350 stacks down. 2 of the stacks came up without issue, 1 of the stacks had 2 switches that didn't join the stack. That stack had to be rebooted manually and then all switches came up in the stack. 

I will be upgrading to 9.27 in the near future. 

[MRCUR]

@bholmes12 9.27 seems to be much better at recovering from this issue. While in the past I had to reboot stacks where they were reporting incorrect membership after power outages, on 9.27 this alert has been resolving itself or not happening at all. 

[bholmes12]

Unfortunately 9.27 has not resolved the stacking issue I am having. 

OSPF goes up then down and switches in the stack without their own uplink can fall out of stack. 

I have a maintenance window scheduled for this weekend and will engage support so hopefully they can see this issue happen in real time. 

[Mr_IT_Guy]

Soo I took my network to 9.27 per the suggestion of a tech for another unrelated issue. When we pushed it out to one of our networks, it broke my routing. Ended up rolling back to 8.10.

[MRCUR]

@Mr_IT_Guy I think routing not working is listed as a known issue for 9.27 now. It's been fine for me on MS350's though for both local interfaces and OSPF. 

[bholmes12 · Accepted solution]

@Mr_IT_Guy- Upgraded my MS350 stacks to 10.14 a few weeks back and so far they have been solid. The original issue was nmap scan's would cause some switches in a MS350 stack to become unreachable, they would also no longer forward traffic. The only way to bring the switch back online was to reboot it.

Here is the reported fix from the Meraki engineering team.

"The fix was to implement a rate-limiter in hardware to avoid the CPU from being hampered by the scans."

[TomatoSteve]

8.10 with physical stack cables upgrade ASAP. I do find it strange that 8.10 is so called stable release when it has so many bugs vs the so called beta versions that address most the issues. 

[DarrenOC]

This has certainly got me worried now. Recently deployed five stacks of approx 6-7 switches per stack and we've already seen some of these funnies occur. Will have to look at upgrading the stacks but the upgrade process withou a trusty blue console cable worries me!

[pbrischetto]

I'm also having issues with stacked MS350 switches however it's a different (yet equally concerning issue).

I have 2 stacks; Stack 1 and Stack 2. Stack 1 has 2 switches and stack 2 has 3 switches. Between the 2 stacks I have Link Aggregation (LACP) setup using 2 x 10Gb fibre. These connections are split between the 2 switches in stack 1 and are plugged into switches 1 & 3 in stack 2.

So connected as below: 

Stack 1 / Switch 1 --- Stack 2 / Switch 1

Stack 1 / Switch 2 --- Stack 2 / Switch 3

Stack 1 / Switch 1 then connects to my MX.

I then take my laptop and plug it into Stack 2 / Switch 2 and run a constant ping to both the MX and Google. 

If I then pull power to Stack 2 / Switch 1 I lose pings to the MX & Google for 3-4 pings. Being a LACP Link Aggregation I would expect to lose 1 ping at the most. The other strange thing is that I also lose 3-4 pings when Stack 1 / Switch 1 comes back online.

Same thing happens also if I kill power to Stack 2 / Switch 3 or Stack 1 / Switch 2.

Logged the issue with Meraki support who have run me through numerous firmware updates but none of these have resolved the issue. They have now escalated it to their engineers.

Thankfully the switches are still in my lab and are not due to be installed until later this year but I don't think I'll be installing them until this is fixed.

[bholmes12]

What version of firmware are you on? 

I'm taking my entire environment to 9.27 this weekend. 

[DarrenOC]

8.10

[MRCUR]

@pbrischetto We also had similar issues when using cross-stack LACP uplinks until 9.27. 

[DarrenOC]

Same here! Glad it's not just us suffering these issues

[pbrischetto]

Running 9.27

[pbrischetto]

Known issues in MS 9.29 firmware

Known issues

• Layer 3 routing breaks when adding L3 active OSPF enabled interface
• Stacked switches may not process link-state changes on aggregates

I'm pretty new to Meraki so for those that have been using Meraki for a while, how long does it usually take Meraki to fix known issues like this?

Thanks,