MX firmware issues on "brand new" MXs

[RaphaelL]

Hi ,

I'm facing a weird situation 100% caused by bad inventory management on our end. We bought thousands of MX68 couple years ago , stocked them in a warehouse and slowly deploying them. We are now at a stage that some MXs have NEVER been deployed after couple years and obviously are running an old 2018 firmware.

That old version ( 14.xx ) CANNOT upgrade to 19.X or 18.X without going to 16.X. If you simply upgrade from 14.XX to 19.XX it will reboot in loop and fail the upgrade every time ( without letting you know that the upgrade cannot complete 💔 ).

Opened a case , support suggested creating a dummy network with version 16.16 and doing that firmware upgrade process : 

14.34 -> 16.16 -> 18.107.5 -> 19.1.9

You cannot add more than 1 MX per network and I have hundreds to upgrade. We don't know the firmware on the boxes so, we can't really take a chance and ship a MX to a site and expect ZTP not to fail.

If only the dashboard could check the firmware version and do that step upgrade process by itself 😉

I don't see any easy solution. Every option is time consuming.

[alemabrahao]

---

@RaphaelL wrote:

Hi ,

I'm facing a weird situation 100% caused by bad inventory management on our end. We bought thousands of MX68 couple years ago , stocked them in a warehouse and slowly deploying them. We are now at a stage that some MXs have NEVER been deployed after couple years and obviously are running an old 2018 firmware.

That old version ( 14.xx ) CANNOT upgrade to 19.X or 18.X without going to 16.X. If you simply upgrade from 14.XX to 19.XX it will reboot in loop and fail the upgrade every time ( without letting you know that the upgrade cannot complete 💔 ).

Opened a case , support suggested creating a dummy network with version 16.16 and doing that firmware upgrade process : 

14.34 -> 16.16 -> 18.107.5 -> 19.1.9

You cannot add more than 1 MX per network and I have hundreds to upgrade. We don't know the firmware on the boxes so, we can't really take a chance and ship a MX to a site and expect ZTP not to fail.

If only the dashboard could check the firmware version and do that step upgrade process by itself 😉

I don't see any easy solution. Every option is time consuming.

---

To be honest, I don't see an easy solution for your situation.

What I would do in your position would be:

Create a dummy network with firmware 16.16 (which I don't think is possible).

Add one MX at a time, upgrade it through the steps.

Then move it to production.

[Ryan_Miles]

One thought would be use API or bulk creation tool/csv upload to bring in all the MXs and place them into their individual networks. CSV could be as simple as list of serials and generic network names (ex. 1, 2, 3, etc).

Create templates for each of the firmware versions. You'd need Support to help set those old versions. Then bind all the networks to the first template to have them upgrade. Once done rebind to next firmware level template. Etc.

Example, three templates set to 3 different firmware versions. Support would need to set the older versions as those are no longer public facing/selectable.

Org firmware view with example networks 1, 2, 3 bound to each template.

My thought is bind all networks to template v.16 let them all upgrade. Once finished rebind to template v.18. Once finished bind to template v.19.

[RaphaelL]

Exactly , at the moment this is our plan.

It's painful , but I don't think there is a short term solution for that.

[Brash]

A bit of a painful situation.

I had the same thought as @Ryan_Miles - using the API to create temp networks to assign the MX's too will save a bunch of time.

One other thing is you may want to setup a different org to do it in? I'm not sure how many networks an organization can handle before it starts grinding in the dashboard, and you probably don't want to impact the management of your prod networks.

[PhilipDAth]

Horrible!!!

Knowing that you are a scripting genius ...

Script flow:

* Provision MX68 as normal.  If it comes online within 10 minutes, end.

* Create a random temporary provisioning network (like a tmp file) set to 16.16

* Move MX to temp random network.  Wait 10 minutes.

* If it does not come online, flag for human review

* Set to 18.107.5.  Wait 10 minutes.

* If it does not come online, flag for human review

* Move it back to the original network

* Delete tmp network

I'm also thinking you must be able to identify the serial number of all of these units (or at least identify the serial number of every MX68 not currently in a network).

You could amend the first step to check if the MX has one of these serial numbers, and then, when all is finished, remove that serial number from the list (so it is never processed again like this).

[PhilipDAth]

Another option.

During the school/university holidays, employ a bunch of students.

Create a temporary network for every single MX (you could even do this in a seperate org dedicated for this process) for 16.16 and 18.107.5.

Setup a 48 port switch for the 16.16 networks, and another for the 18.107.5.

Have the students first plug the switches into the 16.16 switch.

You'll need a script to automate the process of moving the MX to the 18.107.5 network.

Have the students move the MX to the second switch (doing up to 48 MXs at a time).

Put the MXs back into their boxes.