MX64 & MX65 Firmware Upgrades Hanging / Missing Configurations

[JoelKelley]

Two weeks ago we started seeing issues with MX64s and MX65s that so far Meraki has no explanation for.  We unbox the MX, connect it to the internet and we may see one of these behaviors:  

1. The MX firmware upgrade process is taking an hour or longer to complete.  Normally this process takes about 20 minutes.  
2. The status light on the MX indicates the firmware update is complete (goes from blinking white to solid white) but the MX is missing its VLAN configuration.
3. The status light on the MX never stops blinking.  The MX has internet connectivity and cloud connectivity (if only partial) but the MX is unable to pass inter-VLAN and VPN traffic.  

These issues are about to bring to hault a massive MX deployment to one of the largest C-Store chains in the US.  We have support cases open under two organizations for these behaviors. For what it's worth, the customer orgs that are having these problems are on n82 and n219.meraki.com.

If anyone else has experienced any of these issues we would appreciate you letting us know.

[ChrisSmith]

I have seen this behavior in some of my MX65's. The Cloud Dashboard says it's up to date but the device UI says it's only at 25%. I was successfully able to bring one device back to solid white LED after holding in the reset button for approximately 10 seconds or until the LED turned solid amber.

Once it reloaded, the LED was solid white and the device UI no longer had the 25% upgrade firmware status. This was done in the office. I have not yet tried that in the field.

[Warren_Moore]

Did you ever get a resolution or explanation of this issue? I see it was posted in 2018, we are experiencing this in 2019 so it is a bit concerning.

[JoelKelley]

Sorry for the late reply. While Meraki stepped up and made sure we received MX65s that didn’t exhibit the problem (they pre-tested every MX65 before shipping) they didn’t specifically disclose the problem. Generically, they attributed it to a malfunctioning component part within the MX from a specific manufacturer.

[swifty]

Joel - can I DM you for the Meraki case # please ?
We have what appears to be a similar issue, and when I asked Meraki tech support if it was related to this post they couldn't say as I didn't give the ticket no. for him to compare with.
The behaviour we see is the dashboard tells us the units are on 14.42 but Meraki tech say this has failed and they are actually on 12.24, and the dashboard actually is reporting the 'desired' not the running version.
The units go offline for 8-10mins roughly every 2.5hrs.
My understanding is that they never achieve their initial image download from the dashboard so try continuously rather than using the default schedule (weekly?)

[swifty]

Update, we are told (Meraki TAC) this relates to a VLAN tag on the WAN configuration. Apparently an old issue seen strips the VLAN tag from the port config. We are seeing this. When you first connect a unit to the WAN and it connects to the dashboard, then tries to download firmware, if you look at the Uplink config in the dashboard, the dashboard has stripped the VLAN tag from the link. We have been told to add this to the dashboard before the next reload/download happens and it should go fine. What appears to be happening is the VLAN tag gets stripped, the MX can no longer see the dashboard and it recovers itself by rolling back.