New MX 18.107 stable release candidate firmware: many fixes but RIP speedtest

[cmr]

Security appliance firmware versions MX 18.107 changelog

Bug fixes

• Fixed an issue that could result in DHCP leases not being provided by MX84 or MX100 appliances after upgrading to MX 18.1.05 or MX 18.1.06.
• Corrected an issue that could result in a loss of AutoVPN connectivity for an organization until device reboots had been performed, if the organization had previously been created through the organization split process with Meraki support.
• Fixed an issue that resulted in all HTTP traffic failing when HTTP content caching was enabled. We recommend leaving this feature disabled in all cases until it can be formally deprecated.
• Performance improvements
• Fixed an issue that resulted in MX appliances operating in VPN concentrator mode not forwarding IPv6 traffic received via AutoVPN to IPv6 addresses accessible through its WAN subnet.
• Corrected an issue that resulted in being unable to access the device local status page from MX95, MX105, MX250, and MX450 appliances when SGT was enabled.
• Resolved a rare issue that could result in ports configured for 802.1X port authentication with MAC Authentication Bypass to get stuck in a fail-closed state after a reboot occurred.
• Fixed a rare issue that could result in not having connectivity to all non-Meraki VPN peers when many non-Meraki VPN peers were configured.
• Corrected an issue that could result in source-based routes not taking priority over network default routes.
• Stability improvements for Z3(C) and vMX appliances
• Corrected an issue that resulted in traffic shaping not being applied for non-Meraki VPN traffic.
• Fixed an issue that resulted in​​ fragmented packets that were received by an MX appliance and transmitted across AutoVPN getting fragmented after encryption took place.
• Fixed an issue that resulted in wireless clients experiencing degraded performance for HTTP and HTTPS traffic when content filtering was enabled.

Legacy products notice

• When configured for this version, Z1 and MX80 devices will run MX 14.56.
• When configured for this version, MX400 and MX600 devices will run MX 16.16.9.

Known issues

• After making some configuration changes on MX84 appliances, a brief period of packet loss may occur. This will affect all MX84 appliances on all MX firmware versions
• Due to an MX 15 regression, the management port on MX84 appliances does not provide access to the local status page
• MX appliances will now properly validate that DBD packets conform to the appropriate MTU size. If the MX’s OSPF peer has an improper MTU configured, it may cause the OSPF adjacency to fail to properly form. The updated behavior properly conforms to RFC. Please ensure these settings are properly configured on any MX’s OSPF peers to avoid disruption after upgrading to MX 18.1.X.
• Due to an MX 17 regression, RADIUS messages that transit across AutoVPN may fail to be routed correctly.
• There may be an increased risk of encountering device stability and performance issues.

Other

• Due to its limited value as a troubleshooting tool, the speed testing tool has been removed from the local status page.
• Made the content filtering system more aware of system state issues (such as system time not having been set through NTP yet) that would cause content filtering lookup requests to fail. This may make web traffic more responsive in certain edge-case situations.
• Secure Group Tags (SGT) will now be supported on Z3(C), MX64(W), MX65(W), MX67(C,W), MX68(W,CW), MX75, MX85, MX95, MX100, MX250, and MX450 appliances.

[CptnCrnch]

Well, I can perfectly live without the speed test from the local status page. 🙂

[RaphaelL]

I find this pretty scary : 

• Due to an MX 17 regression, RADIUS messages that transit across AutoVPN may fail to be routed correctly.

Any details about that ? 

[thomasthomsen]

Yeah, I have also been keeping an eye on that, so far, I have not seen the issue, but its just strangely specific.

Perhaps its only when the MX does dot1x, and not equipment connected to the MX (just speculation).

[RaphaelL]

Jesus... What happened in that version !!

I'm hitting +100Mbps in download versus MX18.106 and older.

Is it time for to call my ISP and upgrade to 1Gbps?

[LeoTran]

Hey Raphael,

Can you do a ping test via Site-to-Site VPN? Upgraded my Z3 yesterday to test but encountered unstable and very high latency in Auto VPN.

[mrki]

Same issue with VPN. It's highly variable ping going into seconds and occasionally timing out.

[LeoTran]

Thank you. Finally there's someone understood me.

My Z3 has been on 17.10.5 since it was announced. No issue so far. I've been holding off 400 networks to find the actual stable version before upgrading them.

https://community.meraki.com/t5/Security-SD-WAN/New-MX-17-10-5-stable-release-fixes-DHCP-issue-and-m...

[mrki]

I downgraded to 17.10.2 and 30h later it's still good.

[Brad_AgVantage]

we have been having horrible speed issues with remote users.  I appears it has been since our routers started upgrading to 18.107 too.  Seeing ping responses all over the place.  I also have opened a case with support.  We initially thought it was sql performance issue, but after breaking down all the noise that you get from users and customers, and focusing in on who really was having issues it was pointing more and more to network issue.  We had several remote locations that were on 18.107, but our hub was not and we were still having issues.  We upgraded the hub to 18.107 and still same problems.  We even rebooted devices with no luck.  Am going to downgrade some routers tonight to see if that helps.

[RaphaelL]

Upgraded from what version ? What kind of devices ?

[Brad_AgVantage]

we had Z3, Z1, MX64, MX80.  most were at 18.105 and went to 18.107.  some of the devices were seeing ping ranges from 35 ms to over 1800 ms.  We had several applications that were extremely slow since the the upgrade.

[Dave709]

Z3's here and a MX250 Hub.  18.107 caused major issues.  See this ping response!

Pinging 10.59.97.249 with 32 bytes of data:
Reply from 10.59.97.249: bytes=32 time=1682ms TTL=60
Reply from 10.59.97.249: bytes=32 time=3423ms TTL=60
Reply from 10.59.97.249: bytes=32 time=3168ms TTL=60
Reply from 10.59.97.249: bytes=32 time=2440ms TTL=60
Request timed out.
Reply from 10.59.97.249: bytes=32 time=1823ms TTL=60
Reply from 10.59.97.249: bytes=32 time=2578ms TTL=60
Request timed out.

Ping statistics for 10.59.97.249:
Packets: Sent = 8, Received = 6, Lost = 2 (25% loss),
Approximate round trip times in milli-seconds:
Minimum = 1682ms, Maximum = 3423ms, Average = 2519ms

Rolling back to 17.10.2 fixed issue for most, but didn't work on all.  Seems the rollback fails often.

[Silentshooter]

Highest I have ever seen with the Built in throughput test on a MX67, so you might be onto something.  Does seem this FW has increased throughput

[RaphaelL]

Well after I upgraded my home link to 1G/1G , I'm hitting even higher upload with this firmware : 

[LeoTran]

I didn't mention the WAN throughput because it's not a problem, but the Auto VPN tunnel. 
Anyways, I spoke to Meraki Support on a different issue on the other day, and I brought this strange behavior, we took some captures, here is what we have found when executing ping.
+ Meraki to the Internet is fine.
+ From an endpoint sitting behind Meraki MX/Z to the Internet is fine.

+ From Meraki to Meraki (both WAN & inside IP) is fine.

- From an endpoint sitting behind Meraki MX/Z to one of the Meraki spoke is unstable.

- From an endpoint sitting behind a Meraki spoke to another endpoint sitting behind another Meraki spoke is unstable.

He downgrade the firmware, the issue is gone. He said he would check  and monitor with other peers because it's recently released and not many customers have upgraded yet.

[Dave709]

Exactly the same issue in our organization!!  Upgraded 300 clients to 18.107 and the network degraded to unusable! Opened a support case but they have no reports of issues as it's new.  I rolled back all 300 to 17.10.2 and most seemed to resolve while a handful failed the rollback and still have super high ping responses and timeouts.  I really hope a fix if determined soon.

[thomasthomsen]

Im really curious about this one : 

• Made the content filtering system more aware of system state issues (such as system time not having been set through NTP yet) that would cause content filtering lookup requests to fail. This may make web traffic more responsive in certain edge-case situations.

How does that "show" do anyone know ? Do you get the "blocked" page, or what happens when the content filter lookup fails in this scenario ? - Only asking because I have ONE customer who has some strange experiences when "surfing the web" that seems to disappear when content filtering is disabled. 

This customers experience is that they just get a "blank white" page, without content, then if you do a reload of the page, then it works fine.

Happens on every page, not every time, no seemingly pattern

[Quimax]

I've had that recently with my own system. Turned out it wasn't the Meraki at all, but the uBlock Origin extension in Firefox. When I turn that off, those pages work again.

[Fabian1]

Not sure, if we are the only one, but we where just facing problems with the update to 18.107 on a VMX on AWS. Don't know what caused the problem, but we had to reboot the appliance, otherwise the machine wasn't reachable. We've never had something like this with an update, so hopefully it's not a bigger issue.

[JeremyR]

We've experienced that Z3s upgrading to MX 18.107 (from 17.02.10) are unable to pull a DHCP lease on port 5 (the PoE port) on the Z3 appliances post-upgrade. Rolling the impacted appliances back to the prior release (17.02.10) fixes the issue, and re-attempting upgrade to 18.107 reintroduces the same behavior on Z3s with DHCP clients on port 5 only unable to pull a lease once again. Other ports (1-4) are curiously not impacted. Opening a support case on this, but just warning the community of this issue.