Meraki Firmware MX 16.16 problems - MX 100 and MX 64

[TB_A1A]

Hi,

After upgrading to Firmware 16.16 all our Groupe Policy and Whitelisted stop working. Look like it's using Normal Policy even assigning devices. Also, I found that between VLAN some problem as well.

 Anyone experiencing these problems ?

I contacted Meraki Support by email because phone call doesn’t work. Waited ½ day on hold. No one answer…

Rollback it’s not good idea since the 16.16 will be forced.

[Dunky]

@TB_A1A Do you know if any other models are affected - MX67, MX84 or MX85?

Have you heard back from Meraki support yet?

Has anyone else had this same issue?

We have segregated factory subnets and are reliant on inter-vlan L3 rules and whitelists applied by Group Policy on the VLANs

[TB_A1A]

Have you heard back from Meraki support yet?

Yes after long waiting on the call.

Has anyone else had this same issue?

From Meraki Tech no other customers report this problem but it's a new Firmware Feb 28,2022. That mean probably no one did upgrade yet. Strange thing happen to MX64 tested yesterday and same problem with MX100 ; Groupe policy, whitelisted not work... 

I was on WebEx with Meraki Support tech and even him, he didn't understand.

For now we are in full production, I had to rollback to 15.44. All working with no problem.

[TB_A1A]

Hello everyone,

 Just give you guys some update on the situation, with MX64, I don't know what happen but it's now working with Groupe Policy and Whitelisted using new firmware 16.16 without changing anything. I spoke with Meraki Support today and they still don't know... 

As for our MX100 it is installed at our HQ  I will leave for now with old 15.44 firmware for few weeks. See anyone has same problem or not...

If you want to upgrade to 16.16 even it is a release candidate, I recommend to test few unit before updating your entire network.

[Jose_Dias]

Oh my God! Everything is working and nobody knows why!

Seems something from the "BackEnd". Happen to me.

This kind of things worry me so much.

[CharlesIsWorkin]

This response got me the lolz.

[MSchwark]

We have updated a virtual vMX100 and an MX64 to version MX16.16 successfully.  Not sure that helps.

[CharlesIsWorkin]

Subbed! Hope there is a resolution. I wasn't feeling so brave as to update just yet so I thought I'd pop in the forums.

[JDomagala]

What is the support take on that? @Meraki was the issue debugged or reproduced in lab? 16.16 was supposed to be a solution for major issues we have in a vast global network, I think community needs details fast.

[TB_A1A]

Nothing, Meraki didn't help and no idea what happen. The support person from Meraki didn't believe me the cause was by new Firmware 16.16 until he insisted to downgrade back to 15.44 and I did it. Back to normal with zero problem...

Some how next day all our MX64 work again. I contacted Meraki Support and no answer... My guess is probably a lot of people contacted about this and programmer fixed without telling anyone ???

[CharlesIsWorkin]

Wierd! I guess that's good that it's working now?

[Dave2000]

MX100 is working only because it was rolled back. MX64 started working the next day without. It did not affect all of the MX64 either. Strange one. 

[Dunky]

I've just done some testing on an MX67 and both the whitelisting (via a Group Policy applied to the VLAN) worked as did inter-VLAN L3 firewall rules.

[TB_A1A]

Do you have MX100 ? if yes any luck ?

[TB_A1A]

I forgot, I have 2x MX 100 running master and slave.

[Dunky]

Aha, you could break the HA and use the slave as a test in another network then?

[TB_A1A]

Only 2x MX100 at HQ office.

[Dunky]

Sorry no - just a test MX67 and a switch at home I use for testing things

[MSchwark]

We experienced an issue after updating to MX16.16 where the appliance would reboot/go offline about every 55 minutes and take about 20 minutes to come back up.  We experienced this over 4 hours before rolling back, successfully.  We were able to successfully update our software MX device in the cloud but our physical MX100's failed at three sites.  I have an active ticket open with Cisco.

[MerakiManiac]

MSchwark,

Did you ever get a solution from Cisco on your physical MX100's?  I have an open ticket as well and have yet to get it upgraded.  Thanks. 

[Dave2000]

 MSchwark

Thats even worse than our issue. Did you resolve it yet?

[cmr]

We have three HA pairs of MX100s that have been running 16.16 since a couple of days after release.  We do only run the enterprise license, perhaps something in the advanced feature set is the cause?

[MSchwark]

Interesting.  The timing seems to be very consistent.  I tried a single site tonight to see if maybe doing all of them relatively close together may have caused an issue with our point to point vpn's.  Same issue.

[MSchwark]

We have not and attempted our second update tonight.  Same issue and rolled back to MX15.44.  We did successfully install update on vMX100 and MX64.  All of our MX100's have had the same issue.

[cmr]

We have three MX100 HA pairs that have been running 16.16 for a good few days, but we upgraded from 16.15, perhaps it doesn't like the jump from 15.x?

[TB_A1A]

Hello, did you call Meraki Support about this ?

For me, the tech support told me to try again but I don't want to risk and it's our HQ...

[MerakiManiac]

Link to Reddit Meraki MX Update 16.16 / Broken SFP : meraki (reddit.com)  We are experiencing same issue.  We have Cisco brand SFP's connected to Nexus 7K.  As soon as we introduce the warm spare (also on 16.16), the primary WAN interface connected via the SFP begins to bounce until it stays down/down.  Manual shut/no shut on the Nexus to bring it back up after shutting down the warm spare.  

[MSchwark]

Interesting.  We are not running any SFP ports on our Firewalls though and it appears to be affecting our internet ports only.  Digging into the way back, I have seen something similar when we had an issue with the smart jack (timing maybe).  I wonder if I put a switch between the smart jack and the firewall if I get the same issue.

[MSchwark]

We also have had an issue with MX16.16 on MX100 devices being unstable.  We have had an experience were the device will reboot every 55 minutes for about 10 minutes and then repeat.  We have had to rollback twice.  We will be scheduling an update with Meraki on our next attempt.

[TB_A1A]

We are going to wait for next version as long as we could. Look like to many people having different issues...

[Make_IT_Simple]

I think your MX was failing to upgrade to 16.16 that's why it was rebooting. This is a known issue with MX100 if there is an upstream device using an SFP or media converter. This issue was fixed in 17.6.  

[MSchwark]

The more I think about it I bet there is an upstream SFP port.  Thank you.  I may try and update to 17.6 and see what the results are.

[Make_IT_Simple]

Please give it a try and update the post. good luck 

[MSchwark]

Updated one site to 17.6 and it's been over an hour with no issues.  Looking promising, Thanks!

[JDomagala]

Do you monitor perfscore on that MX? I assume it was MX100. Known 17.6 bug is "significant" VPN throughput capabilities decrease for MX100 and MX84. I wonder what that means - how much is significant 🙂 Asked Meraki but no response for now.

[MSchwark]

It was an MX100 and I don't since it is one of our smallest (in sqft) offices.  It was our test site and we have two other, much larger, sites to update.  I will take a look and see if there is anything worth reporting to the community.

[MSchwark]

Do you by chance have a link to this bug or something I can read more?  Thanks

[cmr]

@MSchwark it is in the release notes, available either here in the community, or on the dashboard.  Unfortunately it doesn't say more than that...

[ShawnWillard]

We have the same problem with 16.16 using SFP upstream ports.  A little leery to upgrade to 17.6 with the VPN speed issues.  Will hold off for a few weeks and see if anything changes with 17.6.

[redsector]

No problems with our MX84 after upgrading.

[Armelin]

A lot of problems with NBAR, Layer 7 Firewalls. They should find a way to fix it and not to ask for feedback after confirming this firmware as a stable release.

[Varlaks]

I have the same issue. After upgrading to Firmware 16, all whitelisted device will follow the default Layer 7 filtering. Even if i set a Policy for that machine it will still follow the default Layer 7. We rolled back to 15 before but now we dont have the option to roll back. Case created with Meraki support but no solution so far. We now just get call everyday about blocked websites and frustrated users.
No solution yet. Firmware 16.16.1

[MSchwark]

I was able to update to 17.6 which fixed the issues with the upstream SFP port.  This is currently working for us.

Also, not sure but the rollback option for Security devices is a bit different. I believe you have to go to Organization > Monitor > Firmware Upgrades and then on the overview tab you have to look under most recent changes, go back to where you updated to 16.16.1 and then select the rollback icon from there.  I know trying to schedule an update and targeting a previous version will not allow you to select next.

[Varlaks]

Meraki support helped us roll back to firmware 15 as we cannot see firmware 15 ourselves. Was told not to upgrade till they have a fix. Staying on 15 now. 

[FIRMWARE]

Is there news for the pending firmware upgrade? i'm also waiting for a fix.

[Dave2000]

We (TB_A1A) went back to 15 after trying the newest 16 last week. Just so many issues still

[Varlaks]

They told me to test on Firmware 17 and told it was sorted. I had to wait till the weekend window as we dont have test lab. Tried and still same. Layer 7 default setting will apply regardless of the machine Policy status. Whitelisted or not the Layer 7 will still apply. So those users on different setting then default cannot access alot of their websites. The NBAR feature does alot more blocking than the orginal Layer 7 setup. Still no fix and still on Firmware 15. Probably will be for another year.

[FIRMWARE]

Hi,

Anyone here has data on the Firmware ver. 17? I'm still stuck on ver 15 since the ver 16 has issues which we rolled back from previous ver. 15.

[MSchwark]

We updated several MX-100 and one vMX100.  the vMX100 (virtual) updated to 16.16 without issue and is currently still on that version with a scheduled upgrade scheduled.  The issue we had was with our physical MX100.  When upgrading to 16.16 all of our MX100's would drop out, consistently, every hour for about 10-20 minutes, consistently.  We were able to roll them back successfully.  I tried several times but eventually updated them to version 17.6 which resolved the issue.  The issue had to do with an upstream SFP connection at the DMARC.  We have been running version 17.6 without issues for several months.  We have not had any bandwidth issues on 17.6 and will upgrade to 17.8 this next week.