Notice of Performance Issue: MX75/85/95/105/250/450 Models with MX 18.211

[haupt]

Users of MX75/85/95/105/250/450 devices on firmware MX 18.211 may encounter performance degradation as a result of a known fault. Specifically, traffic shaping settings marked as "High" or "Low" priority could mistakenly discard traffic traversing VLANs or AutoVPN. To resolve this issue, please modify the relevant traffic shaping settings from "High" or "Low" to "Normal," or alternatively, revert to a previous firmware version. For support with either of these measures, please contact Meraki Support for assistance.

[RaphaelL]

Hi ,

Your notice is different from the changelogs : 

Known issues - may 19th update

• Due to an MX 18.211 regression, networks that have traffic shaping rules configured with a "high" priority may incorrectly drop traffic being routed between VLANs or AutoVPN.

Is it possible to specify the MX models affected in the changelog ?

[GiacomoS]

Hey @RaphaelL ,

It is the same issue. We are working on aligning the various channels, but to our awareness at the moment this is impacting only the models mentioned above. 

Many thanks!

Giac

[RaphaelL]

Great thanks ! Only mentionning it because I was about to cancel or postpone our MX68 migration to MX18.211 , but seems like they are not affected by that issue. phew. 😎

[BHC_RESORTS]

Was banging my head against the wall trying to figure out why some devices weren't connecting over a site to site VPN. This seems to discard traffic that isn't even associated with traffic shaping rules.

@haupt Does this bug apply if the default traffic shaping rules are applied? Also, is this only on the global/WAN traffic shaping, and/or is it SSID based shaping?

[GiacomoS]

Hey @BHC_RESORTS ,

I believe it should be just with custom rules, as you can't set the priority on the default ones. 

The behaviour is specific to the MX platforms mentioned at the top, so SSID traffic shaping should not be affected.

Hope this helps!

Giac

[cockroach_c]

Going to find out in the morning if our downgrade solves the issue. We updated to MX18.211 the same morning issues started being reporting (just one business day ago). It appears that some cross vlan traffic is dropping. The results almost seem random. For instance, some PCs can reach certain file shares, others can't. Some resolve DNS, other's don't. 

I did try changing our traffic shaping policies to normal (a few were at high, others were at low). So far, this has not resolved the issue, but I also haven't had the downtime to reboot our network stack. Here's to hoping the firmware revert works... 

[GiacomoS]

Hey @cockroach_c ,

Please, let us know how it went, or alternatively move to 18.211.0.1 instead. If you are still seeing unexpected behaviours, please do raise a case through Dashboard so we can help investigate. 

Packet captures of what doesn't work would help.

Many thanks!

Giac

[cockroach_c]

Same issue going down to 18.107.2 per meraki support request. I am trying to see if there is anything I can look at prior to calling meraki with my ticket number again. Trying some basics right now just to reset network connections for some of the test machines. 

It is very strange. The issue started the same morning that the device was updated to 18.211 (we actually have two organizations that updated). We first noticed an issue when connecting to SMB shares. Now the shares sit on one vlan and end users trying to access come from another. So I would think that is the cross vlan traffic we was mentioned as dropping. Am I correct on that assumption? We did notice later that RDP was dropping between vlans, etc.

[cockroach_c]

I might have lied. This might be working with the revert. Reverting all of the troubleshooting on the test pc before this got to the network engineer...... 😆

[cockroach_c]

And everything up and working. The downgrade worked!

[GiacomoS · Accepted solution]

Folks, 

We have released a hotfix, 18.211.0.1, which addresses the problem with traffic being dropped when you have traffic shaping rules with either a high or low priority set. If you are impacted by that, I would recommend upgrading. 

Many thanks!

Giac

[MSakr]

Does thius apply to default rules? we are seeing degradations but no custom rules and we are not getting the option to upgrade the firmware, greyed

[nb12345]

Hello, We are seeing similar issues with using Google's DNS and the mx is showing consistent packet loss since the upgrade to 18.211.2. Not just google's DNS but any others like Quad9, Cloudflare, Umbrella. Is anyone else expirencing this issue? We have MX250's

[GiacomoS]

@nb12345 , could you DM me a link to your Dashboard or the serial number of the MX please, curious to see what's going on.

Many thanks!

Giac