Staged switch upgrade doesn't complete

[Gumby]

Went to start the upgrade to 10.30 today from 9.37.  Put in a timeframe of 4 weeks off so I could do some testing and roll out over the coming weeks.

First switch upgraded via staging, then the entire schedule for the rest cancelled.  Firmware manager now thinks the entire network is at 10.30 so cannot trigger another update, however all but one switch are still at 9.37, including some critical core switches that I would rather not update in the middle of the day at random!

Have put in a support request, but short of rolling back the current test switch that is on 10.30 (which may have unintended consequences on the other switches), any other ideas?

Anyone else have a staged update fail like this?

[Gumby]

[PhilipDAth]

What happens if you try and schedule it again?

When I've done staged upgrades it is usually over several hours. I wonder if there is an issue with doing it over so much time.

[Gumby]

Can't schedule again, as below.  Never had issues staging across several weeks before, and have done it quite a few times.

[Adam]

Support may have to manually schedule it for you at this point

[PhilipDAth]

Although this doesn't help you, Cisco Meraki has just released a new document of best practices for firmware upgrades.

https://documentation.meraki.com/zGeneral_Administration/Firmware_Upgrades/Meraki_Firmware_Best_Prac...

One thing I didn't know is that when you schedule a switch to do a firmware upgrade it first downloads the firmware, and then it waits 20 minutes for other switches to complete their download in the same cycle.

I also didn't realise that when you select to upgrade a switch in a stack it actually upgrades all stack members automatically.

[BlakeRichardson]

Can you schedule an upgrade to BETA and then roll back?

[Gumby]

So one of the switches failed to download the firmware, still can't work out why.

Due to this, it stops the entire staging process leaving the upgrade in limbo.  Only suggestion was a rollback and attempt again.  I can't say this is a good solution if you get most of the way through an upgrade to fail on the last few switches.

[GiacomoS]

Hey @Gumby,

10.31 has been moved to Stable release candidate, so could you try to upgrade to that version and see if it succeeds?

When this type of issues occurs, I would recommend to give us a call as we can push different firmwares to the single devices to try and sort this out.

@PhilipDAth yes! I find the way upgrades are handled in Meraki quite good as they require minimum input from the users... I remember the terrible times of setting up an FTP server on my laptop just to move an image...  We try to do things sequentially, so we would first store the new firmware on all the devices, then install it on the first, reboot and follow with the second and so on. 

Regarding the stack, I think it is a very common recommendation to have all the devices on the same firmware release when you stack them as otherwise all types of mysterious and magical things can happen to it, so when you trigger a firmware upgrade for switches in a stack, all members will try and upgrade to maintain consistency and stability 🙂 I remember Alcatel switches were particularly troublesome in that regard... the pain...! 🙂

Thanks!

Giacomo

[Gumby]

---

@GiacomoS wrote:

Hey @Gumby,

10.31 has been moved to Stable release candidate, so could you try to upgrade to that version and see if it succeeds?

---

As it failed so early in the staging process I simply did the rollback.  When I was free to upgrade again 10.31 was RC so went to that anyhow.

Second time around upgrade went through without a hitch.

Staging is good, but I still believe there needs to be an option to resume a failed/cancelled stage.

[Zilla]

My team had the same issue going from 8.10 to 9.37 last week.

We had two switches left that needed to be upgraded and when the second to last switch was scheduled the final switch said it canceled the upgrade.

The team opened a support case and support had to pin the final switch to 9.37 to get it to upgrade.

Last month we used the feature without issue on two other networks doing the same upgrade. 

Our upgrades were spread out over weeks since we were performing RMAs.  New switches came in on 9.37, then we upgraded anything else not in the RMA over the course of a few nights.  Finally, the Core switches went a few weeks later.

[Zilla]

FYI, we received an email from Meraki support this morning:

Due to an error in our system, last week we failed to upgrade as scheduled the firmware on some of your
devices. The affected network is:

• (network name hidden)  (28 switches completed, 1 rescheduled)

Your network continues to operate effectively. Nonetheless, we apologize for the inconvenience this causes
you. We have rescheduled the upgrades to be completed 4 weeks from the original upgrade date and time.
Upgrade details can be confirmed using the Firmware Upgrade tool under:
Organization —> Firmware Upgrades —> Scheduled Changes A list of all the networks pending firmware
upgrades can be found there.

You can also choose to reschedule as needed. If scheduled upgrades need to be canceled, please reach out
to our customer support team. You can open a case by clicking "Help" in the top right corner of any page, and
then "Get help".

Again, we would like to apologize for the inconvenience and confusion this has caused. Thank you for your
understanding as we work towards correcting this error.

Regards,
Cisco Meraki

If you go to the affected network you can see the switch that is scheduled in the staged upgrade view.

[MFuchs]

Same thing here.

i was told it is a GUI bug and support is on the run to fix all affected networks ...