Z3 Firmware updates need 16.16.1

[JGill]

Check your Z3 networks if you updated to 16.16.    That "Rare" Z3(C) note on 16.16.1  turned out to be 100% of our Z3 (non cellular) devices.     Dashboard looked like a Christmas tree of very rhythmic comm losses for all the devices.  No errors in logs other than Ethernet Carrier change.  

If you pull the device data by API you can see the firmware issue.  'firmware': 'Not running configured version'.  

Bug fixes

• Corrected a rare issue that could result in Z3(C) appliances failing a connectivity safety check after upgrading to MX 16 and falling back to their previous version.

Should read:

Bug fixes

• Corrected an issue that results in Z3, Z3(C) appliances failing a connectivity safety check after upgrading to MX 16 and falling back to their previous version. 

Not obvious unless you filter down to just Z3 units on the overview page.  

[tomas209ca]

I had this bug knock out some z3/c and now will have to do some driving around to get them up. I think this also has issue with any z3/c that is on any firmware 16.x, as i have some that have gone offline but still pass traffic but not manageable via dashboard.  No way to get them up.

[JGill]

Mine were trying and reverting at least every 24 hours.  So might be worth trying to push the 16.16.1.  If they check in might get the version update. I'll know after the weekend if it truly fixes the issue.   But at least after pushing 16.16.1, the version numbers via the API show correct / updated.  

[PhilipDAth]

That is a goody.  Thanks for the heads up.

[JGill]

After monitoring over the weekend, 16.16.1 appears to have returned our Z3 units to a stable condition.  The Christmas tree pattern is gone!    

[JGill]

Well while the 16.16.1 got our Z3 units to stop rebooting every 24 hours.  We now see a new issue where the units just go into never never land for 1 - 3 hours.    Red Blinking light, no local link, just stuck.   They do come back, but have not seen a pattern just yet.   Seems to be hitting at random times at least once or twice a week.   So frequency is less than the 16.16 issue, but impact is much larger to the users.   So unless we get an answer soon, we will be rolling back to 15.44 this weekend.   

For what its worth we are also seeing issues with MR's loosing access to the console, but they appear to keep running.  The Z3's just go dark.   

Does not look like Meraki is having a good month with updates 😞

[Banfield75]

I have same problem with my two MX84 in HA. The underlaying switches looks the same. I have tried 16.16 and 16.16.1. No luck.

[JGill]

Luckily we have not had the issue hit our MX84's (all in HA). With ~ 237 network locations its going to be an ugly day if that happens!  For now we have rolled our Z3's back to 15.44,  other than needing to bounce 5 of the 70 devices they seem to be doing ok with the exception of one unit.  User we be at that site tonight so we are hoping resetting gear / broadband will resolve the issue.   Bet you can't guess what time we rolled back 😉

[JGill]

FYI,  Had to paper clip the device to get it back online 😞.  Rollback not so clean.  Have had two units we had to factory reset after rollback so far.