MS Public Preview Available: Staged Upgrades

[Minyi]

We are thrilled to introduce the public preview of the improved MS staged upgrades! This new, improved staged upgrades page will automatically appear and replace the previous Switch > Staged upgrades page on the dashboard. 

Check out this new feature announcement post to learn more. 

[Inderdeep]

@Minyi : Excellent, Thanks for the update !

[BlakeRichardson]

Awesome news, looking forward to putting this into action however it would be nice to be able to search for devices when adding them into a group instead of having to scroll through a list.

Thanks!

On a side note having this ability for MR hardware would be awesome. The more control we can have the better. 

[Brennan_Martin]

Hey Blake! We’re definitely giving some thought to ways to search and quickly find switches to add to a group, so I’m glad to hear this came to mind for you too. We have a few ideas in mind to keep making staged upgrades even better, but wanted to get it out to you as quickly as possible because there’s nothing better than feedback from live networks. 

I’m curious, how many switches are you dealing with in the network where you’d like to be able to search for switches to add to upgrade groups?

[BlakeRichardson]

@Brennan_Martin The network that made me think of this currently has 42 switches with more being added next year. Even being able to arrange the switches by name would be helpful.

[Brennan_Martin]

Ok, cool. Thanks again for the feedback, and we’ll look into this!

[cmr]

I'm sad to see that stacks are grouped as one. We've been scheduling half of a stack at a time with MS355Xs with good success. As they are our core at 24/7 sites, we might have to break them now 😢

[Brennan_Martin]

Hey @cmr , I know right? Being able to keep a stack online during an upgrade (kinda like ISSU) is a super cool feature to have. Unfortunately, ISSU isn’t what’s actually happening there and we actually never supported splitting stacks into separate upgrade stages. In fact, doing that has caused problems where new and previous firmware images are mixed in a stack, resulting in forwarding and stability impacts that were not so awesome.

Sorry to kill the mood there, we elected to play it safe on this one. That being said, something like ISSU is definitely something we’d like to look at properly implementing in the future so I’ll put a vote in the bucket on that item. 

Thanks for the feedback.

[GIdenJoe]

A switch like the MS425 which usually is a flexible stack would be the best candidate to support an upgrade where one switch goes down and upgrades before the other.  So it resembles the ISSU on a C9500 Cisco switch.

[Brennan_Martin]

Hi @GIdenJoe  – the physical and flexible stacking logics are the same with regards to firmware upgrades. We’ve tested staggered upgrades on MS425s in the past and they are not hitless. For that reason, staged upgrades across members of stacks aren’t supported.

Convergence of hardware forwarding information should occur within 30 seconds of a failure or in this case a reboot for upgrade, however, this period could be longer depending on the nature of the failure, the definition of full convergence, and the protocols involved. For instance, if it’s the active switch in the stack that fails, protocols such as OSPF will have to re-establish adjacencies and synchronize the database.

[Slobs2]

Instead of playing it safe, why not allow folks the ability to use the old page like others pages allow?  Having to shut down my entire virtual environment to upgrade my MS425 stack is pretty unreasonable. This is a complete loss of redundancy. I’ve updated the stack separately for years and not had any issues. This is truly a disappointing change!

[Brennan_Martin]

Hey @Slobs2, I’m sorry to hear we’ve missed the mark for you on this. It was a tough decision but know that contrary to your experience, we’ve had a lot of customers that have had problems when they try to split stacks into separate upgrades. That being said, we also hear that being able to upgrade stacks without interruption would bring a lot of value for customers like yourself so we’ll investigate to see what is feasible. 

[cmr]

Well, I tried out this feature to go to 15.14.2 on a site running 14.32 that had a stack of 3x MS355X-48 switches as the core and an MS120-24P on its own.

The 120 upgraded as scheduled, but the stack crashed with the middle switch being "sort of" connected.  I got a remote pair of hands to pull the power cables on the recalcitrant switch and everything came back up.  This did happen at one time in the past when I did a split upgrade, so I'll try another site and confirm how well it works there.

[Brennan_Martin]

Hey @cmr, I want to make sure I understand correctly - were you able to split switches that are part of the same stack into separate upgrade groups, or were all of the switches of that stack in the same upgrade group and the issue occurred after attempting to upgrade that stack?

[cmr]

@Brennan_Martin the latter, upgraded as one and the middle switch needed a hard reset for the upgrade to complete.  We did have this before at one point when split upgrading this stack, so I thought I'd upgraded a stack that has never been split upgraded to see how that goes.

[Brennan_Martin]

Gotcha! Sorry you ran into an upgrade issue, it’s frustrating when that happens. On the bright side, that’s not an issue with the staged upgrades feature 😁