New MX 18.106 Stable Release Candidate - multiple reliability and performance imporovements

[cmr]

Security appliance firmware versions MX 18.106 changelog

Bug fixes

• Corrected an issue that resulted in intrusion detection rule update events not being logged to the event log.
• Reduced the amount of time before a URL classification request performed by the content filtering service would be considered to have failed. This may resolve latency caused by content filtering in cases when classification requests failed and needed to be retried on a consistent basis.
• Resolved an issue that could result in connectivity checks to Meraki cloud infrastructure that are performed prior to allowing an upgrade to be scheduled to erroneously fail.
• Stability improvements for MX64(W), MX65(W), MX67(C,W), and MX68(W,CW), MX75, MX84, MX85, and MX100 appliances.
• Corrected a rare issue that could result in Z3(C) appliances incorrectly discarding traffic.
• Fixed a rare issue that could result in the WAN interfaces for MX appliances incorrectly transitioning to a down state for a brief period of time.
• Resolved an error that resulted in the VPN status page reporting the incorrect status for VPN connections to non-Meraki VPN peers formed using an FQDN instead of an IP address.
• Corrected a rare issue that could result in MX84, MX100, and vMX appliances being unresponsive after rebooting.
• Performance improvements for VPN traffic for Z3(C), MX84, and MX100 appliances.
• General performance improvements. These are most likely to be noticed in environments where MX appliances are processing a high volume of packets.
• Fixed an issue of incorrect fragmentation handling for IPv6 packets.
• Resolved an issue where an MX appliance could stop responding to LAN traffic when a frame was received with a source MAC address matching the MX’s own LAN port. This issue was most likely to occur in complex switching topologies, or when there were forwarding issues within the network.
• Corrected an MX 18.1.X regression that could result in incorrect power LED behaviour during the bootup process for MX65(W) appliances.
• Fixed a rare issue that could result in disruptions to DHCP in High Availability configurations.

Legacy products notice

• When configured for this version, Z1 and MX80 devices will run MX 14.56.
• When configured for this version, MX400 and MX600 devices will run MX 16.16.6.

Known issues

• After making some configuration changes on MX84 appliances, a brief period of packet loss may occur. This will affect all MX84 appliances on all MX firmware versions
• Due to an MX 15 regression, the management port on MX84 appliances does not provide access to the local status page
• When SGT is enabled on MX84 appliances, any packet larger than 1440 bytes will be dropped. Due to this, we recommend that the SGT feature only be enabled in lab or other non-production environments on MX84 appliances.
• MX appliances will now properly validate that DBD packets conform to the appropriate MTU size. If the MX’s OSPF peer has an improper MTU configured, it may cause the OSPF adjacency to fail to properly form. The updated behavior properly conforms to RFC. Please ensure these settings are properly configured on any MX’s OSPF peers to avoid disruption after upgrading to MX 18.1.X.
• Wireless clients may experience degraded performance for HTTP and HTTPS traffic when content filtering is enabled.
• There may be an increased risk of encountering device stability and performance issues.

Other

• Added firmware support for reporting when the AnyConnect VPN service starts or stops in the Event Log.
• Minor improvements to the reporting of flow data via syslog.
• Slightly reduced the power usage of MX64(W) and MX65(W) appliances. Power savings range from approximately 0.5W to 1.25W.
• MX appliances will now use 64 as the CurHopLimit value in IPv6 Router Advertisement messages.

[thomasthomsen]

Thank you finally .... 

• Resolved an issue that could result in connectivity checks to Meraki cloud infrastructure that are performed prior to allowing an upgrade to be scheduled to erroneously fail.

And Im REALLY curious about this one:

• Fixed a rare issue that could result in the WAN interfaces for MX appliances incorrectly transitioning to a down state for a brief period of time

Do anyone have more information on this issue, because I have a feeling that it is "not so rare" on the MX85 for example.

[JLevine]

Confirmed from Meraki support that "Fixed a rare issue that could result in the WAN interfaces for MX appliances incorrectly transitioning to a down state for a brief period of time" has to do with uplink status changes when utilizing both WAN ports on MX85. Finally!!!

[RaphaelL]

Huge update ! Kinda suprised not seeing these changes ported to MX17

[Lurick]

Not able to test right now but anyone with a wireless MX can you confirm if this fixes wireless client connectivity issues and slowness?

Edit:

I see I was unable to fully read the notes 🙂

[RaphaelL]

Still listed in the known issues , no ? : 

• Wireless clients may experience degraded performance for HTTP and HTTPS traffic when content filtering is enabled.

[Lurick]

No clue how I missed that, lol, thank you 🙂

[RaphaelL]

hehe no worries ! I myself missed some too 😛

[OVERKILL]

So, I see most of my MX's that are running 17.10.2 (stable) have been automatically scheduled to upgrade to this release. This seems to imply that it's a more important update, or some of these fixes are more important than one would initially assume, no? 

[JGill]

So anyone have any insights on this statement?  

• There may be an increased risk of encountering device stability and performance issues.

Between the slow wireless with content filters and that statement, I'm not sure I want to pull the trigger on this forced update release without some additional details.  

[ww]

That is a default note in (almost?) all rc/beta fw 

[LeoTran]

I just tested this firmware with a Z3, there's still a known problem with RADIUS over AutoVPN since MX17.

[RaphaelL]

Do you have more info ( or a thread ) about this issue ? Is it only affecting Z Series ?

[LeoTran]

Hey Raphael,

Not limited to Z series. It affects Meraki Security Appliance model that has a Wi-fi built-in.

I opened a case with Meraki since Jan and the Support Engineer confirmed with me there's a known bug on this.

https://community.meraki.com/t5/Security-SD-WAN/New-MX-17-10-2-stable-release-candidate-firmware-cel...

https://www.reddit.com/r/meraki/comments/zk3w6p/nasty_bug_in_mx_17102_for_mxs_with_builtin_wifi/

https://community.meraki.com/t5/Security-SD-WAN/New-MX-18-105-Stable-Release-Candidate-fixes-for-VPN...

Cheers,
Leo

[sealbon]

It could be fixed in 17.10.4 now.

https://community.meraki.com/t5/Security-SD-WAN/New-MX-17-10-4-stable-firmware-more-performance-and-...

[LeoTran]

Thank you for letting me know.

Just opened a case with Meraki as I couldn't downgrade to that version. Can't wait to test it on my Z3 before upgrading.

Cheers,
Leo

[tcanty]

"Minor improvements to the reporting of flow data via syslog", a couple of our MX's upgraded 10 106 last night, and since then there has been no patterns "deny all" come through, don't suppose anybody knows what the minor improvement was?

[dgander]

As for:

• Performance improvements for VPN traffic for Z3(C), MX84, and MX100 appliances.
• General performance improvements. These are most likely to be noticed in environments where MX appliances are processing a high volume of packets.

Has anyone with a MX100 who has been having the WAN throughput degrade to over 50% after 15.44 tried this firmware? Cisco support says this release FINALLY fixes this issue and wondering if anyone has tried.

[LeoTran]

I saw the performance on the latest patch MX 17.10.4 is way better than this MX 18.106.

[dade80vr]

3 case in these days after MX upgrade: a lot of websites won't load despite not being blacklisted!

Rollback done to v17.10.2 and problem solved

Please Meraki don't force to any stable RC release!!!

[jsphrcross]

Hey dade80vr

I have been having the same issues upgrading to v17.10.4.  I have ticket open with Meraki as of yesterday.  I have rolled back (3) organizations so far and have many more orgs to deal with.  Not only are some sites not loading sporadically, Windows systems are receiving this error in the logs ~ every 10 seconds:  SChannel (event ID 36876) errors in Event Viewer -> Windows Logs -> System     along with systems are not being allowed to report into our Carbon Black AV servers.  It seems the newer firmware/s believe the certs have expired or aren't valid.  I am still collecting data but it seems to affect the MX84's more.  Can you confirm if your issues are on MX84's and/or you see the SChannel errors on Windows systems in your environment/s?

My post here in the community is here.  I have only had once response.

https://community.meraki.com/t5/Security-SD-WAN/SChannel-Errors-on-Windows-Systems/m-p/187260#M43866

[IDV]

Hi sphrcross

I had problems also with the update to 18.106 and even to 17.10.4 specifically with MX84, where the rollback solved it. The problem I faced was that MX was not loading some http pages, apparently because it was redirecting port 80 to 443.

I opened a case but unfortunately Meraki suppport have not yet identified the problem and due to the lack of a window to carry out joint tests they have not been able to confirm it.

I hope this comment will help the community and be aware of this update.

Best,

[jsphrcross]

I am on the phone with Meraki right now working it.  I just shared your information with them.  Thanks @IDV 

[dgander]

Thanks, I sent this into my case as well since they want us to try it. I am still on 15.44 since the bandwidth issue introduced in 16.X and wasn't addressed in 17.x either

[RaphaelL]

Was the MX sending the TCP-reset ?  It would pretty easy to determine with the IP TTL.

Take the TTL of the SYN-ACK from the 443 conversation and compare with the TTL of the TCP reset. 

[IDV]

Hey RaphaelL

It seems that this confirms it.

Your suggestion had a better approach than that of the support ing that assisted me. Thanks.

[jsphrcross]

After ~ 3 hours on the phone with Meraki and sharing the information provided by @IDV , @Raphael, @dgander , @dade80vr  & @dade80vr we have a "bandage" to resolve this until an update to fix it is released.  I was informed that Meraki is aware of the issue and is working on a solution.  No ETA as to when the actual fix will be known.  Meraki also stated that this is only affecting MX84's.  I suspect it may affect other models as well that the setting listed below is available on.

Solution:  Disable Web Cache and reboot router.

Security & SD-WAN -> SD-WAN & Traffic Shaping

Unfortunately a reboot is required for the change to take effect.

I hope this helps anyone currently having the issue.