MX 18.208 release being pushed - Performance improvements added to the 85/95/105

[Mloraditch]

Edit: I can see the full notes now and and am pasting. Original post still below for history of this breaking news 🙂

What's new

• Significant performance improvements for MX85, MX95, and MX105 appliances.

Bug fixes

• Resolved an issue that resulted in Event Log data not being generated for all clients of MX75, MX250, and MX450 appliances.
• Corrected a rare issue that could result in large numbers of routes causing network instability during AutoVPN connectivity changes.
• Fixed an issue that could result in some clients being unable to access WAN resources when MX75, MX250, or MX450 appliances were configured with a PPPoE uplink.
• Fixed an issue that resulted in MX appliances failing to initialize a service required for encrypted communication with Umbrella.
• Resolved a rare issue that could result in SFP+ ports on MX250 and MX450 appliances unexpectedly toggling between up and down states when forwarding incorrectly sized MDNS packets.
• Corrected an issue that could result in IPv6 dynamic prefixes from WAN1 not being installed into the MX routing table.
• Fixed a rare issue that could result in AutoVPN traffic being routed incorrectly after an uplink failover or failback when 1) the MX appliance was configured to operate in High Availability mode (HA), 2) a virtual IP address was used, and 3) a teleworker VPN was configured.
• Resolved an issue that resulted in MX68(W,CW) failing to negotiate PoE power at 802.3at
• Corrected an issue that resulted in Z4(C) appliances being unable to change to a DFS channel after having previously used a non-DFS channel.
• Fixed an issue that resulted in MXs appliances incorrectly modifying the source IP address of ICMP time-to-live exceeded messages when routing them between VLANs.
• Additional changes to increase the robustness of connectivity and self-recoverability for integrated cellular modems.
• Corrected an issue that could result in reduced performance on MX75, MX85, and MX95 appliances when IDS or IPS was enabled.
• Fixed an issue that could result in MX appliances failing to populate the ARP table live tool.
• Resolved an issue that resulted in MX appliances failing to receive an IP address on its WAN interface when DHCPv6 over a PPPoE uplink was used.
• Corrected an issue that could degrade the performance of traffic destined to and sourced by MX appliances when 1) IPv6 was enabled, 2) BGP was enabled, and 3) there were over 1024 AutoVPN peers.
• Resolved an MX 18.2 regression which caused MX appliances to summarize AutoVPN routes advertised through BGP without being configured to do so.

Legacy products notice

• When configured for this version, Z1 devices will run MX 14.56.
• When configured for this version, MX400 and MX600 devices will run MX 16.16.9.
• When configured for this version, MX64(W), MX65(W), MX84, MX100, and vMX100 devices will run MX 18.107.8.

Known issues status

• This list is being reviewed and updated. Many existing issue reports have not been confirmed to affect MX 18.2XX firmware versions.

Known issues

• There is an increased risk of encountering device stability and performance issues on all platforms and across all configurations.

Other

• Control traffic for Meraki authentication will now only be routed out the WAN interface(s). Previously it could be unintentionally directed out other interfaces, based on the routing configuration.

ORIGINAL POST:
Just got a ton of emails on a mystery 18.208 upgrade being pushed. Dashboard says:

Other

• Custom MX 18.208 version
• Please contact support for more details.

The email says Significant performance improvements for MX75, MX85, MX95, MX105, MX250, and MX450 appliances. 

My emphasis in bold, as many of us know the bolded models have not been announced as getting performance updates with this release. The rest of the email info seems to match what we do know about 18.2XX.  A recent forum post says the other models are on the to-do list but no ETA. Anyway, I'm on hold with a confused support engineer and will update if he says anything useful, but wanted to let folks know.

[Ryan_Miles]

It must be in the process of being published. I see the notes if I manually jump forward to that version on the RC release notes page.

And yes it appears to include the multicore improvements that came to the 75, 250, 450 in an earlier release.

[Mloraditch]

Ok I found a shard where I can see that. That's fantastic news. Hopefully the exact numbers will be out shortly. Editing my main post to share.

[RaphaelL]

My MX68CW is able to reach 1Gbps in firewall throughput : 

[andrewberesford]

What did you get with IPS/IDS on. Speedtest for my MX68 with those on I get 270Mbps on 18.107.8 upon upgrading to 18.207 or 18.208 speedtests dropped to 220Mbps. Seems slower

[RaphaelL]

Honnestly I don't use the IPS/IDS module on the MX for many good reasons so I haven't tested yet. This is purely firewall throughput.

[PhilipDAth]

It is scary how hard this release is being pushed.  It has only been out 24 hours or so, and I have zillions of upgrade emails.  I don't think it should be rolled out so quickly.

18.207 was unstable for me.  I am trying out 18.208 now, and after 4 hours or so it is good - but that is not enough testing time.

[Mloraditch]

I don't know if this is accurate. The support rep didn't seem super knowledgeable about the release so I didn't previously mention, but he said something that seemed to indicate there may be an as yet to be announced security or similarly massive issue that required the quick deployment.

[Nomad]

The update email talks about Detailed, live firewall logs can now be seen through a new live tool 
A standard "feature" with other firewall manufacturers that I've been waiting for for a long time with Meraki. I'm really excited to see how the live view turns out and will therefore bring forward the update on one of our MX to tonight 😄
Does anyone else actually feel this way?

[PhilipDAth]

I used it when it first came out several months ago.  It is great!

[Nomad]

Wait, that's been out for a while? Apparently it completely passed me by...
I'm on 18.107.2
How long has this been out?

[jimmyt234]

It came out with 18.207, the previous stable release candidate version

[dcatiller]

What kind of MX did you update? I have MX250 HA pairs, a 95 and a couple of 105s. Curious to see how it went. 

[ToddB]

It was a HA pair for 105s. This is what was being logged. There was no power issues, assumed the box was crashing. It's only passing SD-WAN traffic. 

Feb 14 08:01:19

Sterling DC - Primary

Device boot

Device boot

reason: power event or other

[Nomad]

I updated my HA MX95 last night and it looks like no issues so far. However, I didn't have any performance problems with the MX95 before.

[Frank-NL]

Hi,

Installed 18.208 on MX85. Great performance increase.

But be warned we are experiencing issues with configured 1-to-1 NAT rules. MX dropping random packets as if there is something wrong in the session/translation table lookups

[Nomad]

Good to know, thanks for the hint.

[B523]

We are experiencing the same issues, I've narrowed it down to replies being blocked on stateful connections.  

[BHC_RESORTS]

Interesting, this could have been the root cause of our MS Exchange issues where we saw extreme timeouts and disconnects.

[Produceman]

Had the same problem with our 1 to 1 Nat rules. Had to revert back to 18.107.2.

[KanesRambo]

Same here... 

[Johnny55]

We also had the same issue and had to roll back our MX.

[esmith]

We also had to roll back to 18.107.2 on our MX250 HA pair.  We were having issues with multiple devices that did have 1-to-1 NAT rules.  

[cvsmith122]

How does QA not test for 1:1 nat... Ill be doing manual updates from now on. 

[gregarican]

I wish I ran across this thread before updating to 18.208 this past Sunday. We have a couple of on-prem servers defined with 1:1 NAT on our MX75. They randomly lose outbound connectivity to external hosts/ports, which has caused major headaches with our apps. Most importantly, cloud data backups. Any other LAN host can connect to these endpoints without incident. It's only happening on two on-prem servers that have 1:1 NAT rules in place. Reverting the firmware back after hours tonight. If that does the trick then I'll be relieved!

[gregarican]

After rolling back the firmware these two inside NAT 1:1 hosts are communicating outbound just fine. Previously it was spotty at best. I still am somewhat amazed that this firmware was considered a Stable Release Candidate. Even the most basic beta testing should've revealed NAT 1:1 hosts having connectivity issues. Reminds of this back in the day when I managed IT for a larger corporate call center --> https://rcpmag.com/articles/1999/11/16/microsoft-posts-winsock-hotfix-for-sp6.aspx.  

[cvsmith122]

I was just running into this issue as well, for the last week i have been having random outbound timeouts to external hosts from my internal 1:1 servers. I called meraki and they said oh yeah 18.208 is a release candidate. How would you push a RC to all customers. I am most defiantly frustrated with meraki right now. 

[cvsmith122]

did the roll back last night and my 1:1 servers are working again ! 

[jimmyt234]

We've experienced the same problem and had to rollback to 18.107.x

[Amit_pal]

Thanks for sharing

[ToddB]

Installed 18.208 on a few MX85's with out issues, but the MX105 was a desaster, boxes were unstable and rebooting every 20 minutes or so. Rolled them back to 18.107.2

[GiacomoS]

Hey @ToddB , do you mind sending me a DM with the serial number of the unit? I'd like to get this checked out a bit more.

Many thanks!

Giac

[CptnCrnch]

First time that 18.2xx was running completely smooth on MX85. Looking ver good, especially performance-wise so far!

[dcatiller]

Has anyone tried this on the MX250? I have two sites with MX250 HA pairs and may try it this weekend. I'm only able to push out the update a month, since getting the notice on 2/13. I'm not seeing an obvious compelling reason to rush the update, so I'm a little concerned. @ToddB I have a site with an MX105 as well. Was it rebooting itself or were you having to reboot?

[Nomad]

I updated my HA MX95 last night and it looks like no issues so far. However, I didn't have any performance problems with the MX95 before.

What surprises me is the new firewall live view. Nothing happens when I want to look at some client traffic. Does anybody know what I'm doing wrong?

No matter which IP I take from different VLANS, nothing happens. The wheel spins for a moment and that's it.

Edit:

Oh man, I have a suspicion as to what it could be. We use client tracking through MAC addresses and I bet the live view only works with IP tracking. Unfortunately I can't use that because my MX95 are in a combined network, so I would have to split the networks first. Can someone confirm my suspicions?

However, there is no reference to this in the instructions.

https://documentation.meraki.com/MX/Firewall_and_Traffic_Shaping/Firewall_Logging

[Ryan_Miles]

The live firewall logging is now broken on certain models running 18.208. It's not due to your client tracking mode. I discovered that night one and reported it to engineering. 

[Nomad]

Really? Fantastic Meraki, releasing a feature in one version only to kill it in the next..
Thanks anyway for reporting to support!

[Ryan_Miles]

Well, it clearly wasn't an intentional move. It's been reported to engineering and I'm sure they'll resolve it.

[Holli69]

a few weeks ago I tried 18.207, but is was very buggy and not working for me, because our servers in the DC are not reachable anymore (3rd party VPN between 2 MX's in different Orgs), switched back to 18.107.8 and everything works fine anymore. In 18.207 AND 18.208 the 3rd party VPN tunnel is shown as up in both locations, but no server is reachable via ping. MX18.208 has the same issues. I rolled back again to 18.107.8 and everything is fine again for me (MX85,MX95,MX100->VPN Concentrator MX450)

[tcanty]

Hi holli69, just out of interest was you seeing this VPN issue on both the primary and secondary wan?

We upgraded a few weeks ago to the 18.207 and had no issues with the VPN untill a lorry broke the main fibre to the building, and the VPN was showing up on the secondary connection, but couldn't get it to send any data, downgraded.the firmware, and then it was back working.

[Holli69]

HI tcanty, 

at the moment we use VPN only at primary WAN Connection, secondary is in use for other traffic.

We couldn't send any data as well.

Hopefully in 18.209 or 18.210 it will be fixed.

[aliog]

We are also seeing this issue. We had 19 sites upgrade to 18.208 last night and this morning 5 of them had a problem routing VPN traffic over a Non Meraki VPN. The VPN Tunnel was shown as up both in the Meraki portal and the remote Cisco ASA device but no traffic was able to route over the tunnel. Devices affected were MX67s and one MX68CW-WW. Downgrade to 18.107.2 on the MX67s and 18.107.8 on the MX68CW-WW has seemingly fixed the issue for now. 

[harmankardon]

Anyone using 18.208 on the MX67C? Still stuck on 17.10.9 because of all the issues with integrated cellular in every subsequent release.

[tcanty]

we have 18.207 running on most of our mx67c's, and other then the vpn thing mentioned just above, we haven't had issues with the cellular, and 208 is meant to have further improvements on that side. We aren't going to be rolling to 208 due to the issues with the VPN, and are slowing moving back to the 18.107 versions, however that then breaks the cellular part.

[CptnCrnch]

Another observation: Device Utilization went down noticeable:

[Fabian1]

We also switched back to 18.107, because we have some problems with our 3rd party Zscaler tunnels, which seems to make trouble after 1-2 days.

[GiacomoS]

@Fabian1 , do you have any more details on what kind of problems you experienced? Do you have a case with Support?

Many thanks!

Giac

[Fabian1]

No case

We have a 3rd party VPN tunnel to Zscaler with a default route 0.0.0.0/0 and after a short time the tunnel was still up on dashboard but no traffic went over the tunnel

We have also routes to MPLS (LAN) and a AutoVPN to AWS/vMX, both were fine

With the stable, we do not having these issues

[AaronS]

We haven't noticed any issues with either 18.207 or 18.208. We've been running 18.207 for about a month, and I've deployed 18.208 to a handful of MX68 devices shortly after the update became available. Pushing out 18.208 to all but one of our MX68 and MX250 devices tonight. Our configurations though don't include what others have mentioned here as issues.

[dcatiller]

Are you running your MX250s as HA pairs? Using Meraki VPN? 

Just curious because I have MX250s and will be upgrading over the next couple of weeks. 

[AaronS]

The MX250s are not currently running in HA. We are using Meraki VPN though.

[dcatiller]

Thanks, I appreciate the info. I hope the updates are successful tonight.

[Narmbruster]

FYI, this version is listed as a "stable release candidate", not "stable".  We upgraded our HA paired MX250s early this morning and have had multiple VPN outages and our Master has failed over to the Spare 4 times since 7:30am.  I am scheduling a rollback this evening.  There is also no way to opt-out of firmware upgrades that are anything but "stable".  This has to be a manual selection to reschedule or cancel.  

[dcatiller]

Thanks for the info. 

I've found that you can reschedule the update for up to 30 days. 

I received the first notification on 2/13 and rescheduled for 3/13.

I went back on 2/16 and was able to reschedule for Saturday 3/16, so it looks like a sliding 30 day window. 

I'm just going to keep moving the window until it is safe to install.

[jimmyt234]

You can also cancel it completely, rather than keep moving it.

[dcatiller]

Didn't see that option until you mentioned it. 

You have to go to the 'Scheduled changes' screen, then you have it there. 

I was on the 'Schedule upgrades' screen and it only have upgrade now or reschedule.

Thanks.

[niko79]

I have one user that after the upgrade to Meraki MX 18.208 (MX250, just last night) won't connect to VPN any longer. Had 2 calls with Meraki and the VPN connection won't most often even ask for credentials and based on packet capture fails at step 2 of negotiation. We were able to connect 2 times successfully, but most attempts fail.

Client is on latest Windows 11 (23H2) (updated after it was already not working, this morning)
Meraki MX250 on its latest MX 18.208

Recreated VPN IPSec connection (do not have AnyConnect licenses).

Other VPN users are connecting just fine which points to a client issue but thought I would bring it up here for some resolution?

In the past, this was a MS update that needed to be uninstalled for VPN to work again. Any ideas?

[dreybrouck]

We seem to be having the same issue.  I'm getting ready to contact support to downgrade back to our previous version that didn't have any VPN issues.  Have you found anything else out about this?

[tcanty]

We have downgraded all our mx's back to the older firmware, and the function returns to working. They have updated the firmware notes to say there is an issue with none meraki VPN over a ppoe connection, so seems to be known by them. You shouldn't necessarily need to contact support to downgrade, if you go to the firmware page it lets you "upgrade" to the latest stable version, once selected it tells you are downgrading.

[cvsmith122]

Can can schedule the rollback your self still if your within 2 weeks, Network-> Organization -> Firmware Upgrades - > Click the rewind arrow for the upgrade. Fill out the information and it gives you a window to schedule the roll back, i did mine last night back to 18.107.2 and my 1:1 nat issues are fixed now. 

[cvsmith122]

There really needs to be a opt out of RC and beta builds. 

[dcatiller]

I was able to cancel the upgrade. I was hesitant to apply it until I saw other people using it, so I scheduled it 30 days out. At first I thought all I could do was move the date, but figured out I could cancel it if I went to 'Organization' -> 'Firmware Upgrades' - 'Scheduled changes'. At that screen you will see the scheduled upgrade and will have the option to cancel. Once I did that, it was not pushed again. 

I definitely agree they shouldn't be pushing an RC like this in the first place. They don't make it easy or intuitive to cancel. It would have been much better to have a 'Skip this upgrade' option or something in the first notice. If I want to be a test bed for Meraki, I'll go get the software myself. I don't need it done automatically. That's not a stable way to manage infrastructure. 

[cvsmith122]

Yeah i think this will be me go to action now, especially since it seems we cant trust meraki to QA their builds or to not give us a release candidate. 

[BHC_RESORTS]

18.208 was a disaster for us. Pushed it out last night, came in today to many issues. Specifically, on-prem MS Exchange server had excessive timeouts and interrupted sessions. It would log ASP errors in the event log, and from the client side, OWA would frequently not work, time out, or give random errors. Outlook full application would time out/disconnect. Also noticed websites would time out, then eventually reconnect several times. For example, was downloading a script from GitHub and the webpage timed out, then reloaded, then downloaded 2-3 copies of the file we were trying to download. These were all on MX105s and 95s.

Also had issues with an MX75 after the 18.208 update where VOIP traffic was not working. Rolled back, issues gone.

I would NOT recommend this firmware at this time unless you are very closely monitoring everything - these are just the issues we found briefly using it.

[andrej_ns]

I can confirm that 18.208 broke things for us this morning. We have x2 MX250s in HA and MS250 HA switches behind it. After the upgrade one of the switches uplink shows disconnected and switch became unreachable. We have downgraded back to 18.107.2 and it all started working OK.

[esmith]

Our upgrade was scheduled over the weekend and we had to roll back to 18.107.2 this morning on our MX250 HA pair.  SIP traffic was giving us one way audio if the SIP gateway would register at all.  Office 365 connectivity issues, and Cisco Duo connectivity issues.

[JohnT]

We have an HA pair of MX95's that are having issues with split tunnel in Any Connect.  We pointed our users to an MX67 with the same firmware an it works fine so I don't know if the issue is specific to the MX95, or the HA setup.

[Cole_A]

What a nightmare. MX250 in HA keeps failing over and all kinds of madness ensues. We just took one of the MX250 offline completely to stop the failover from happening. Support acted like everything was fine, it's not.

[Cole_A]

Another issue identified, our esports team reported that none of their Nintendo Switches would connect for their scheduled NJCAA match, and they may have to forfeit if they can't get it rescheduled. The error was NAT related.

[Lauzon_C]

Did you ever get this resolved? I assume you are getting NAT traversal errors? I am having a heck of a time here. I was on MX 18.210, some of our equipment can't go that high. I had one site downgraded get NAT type B. I can go online with wordwide of regional, but if I try making a room I can, but I cannot join with another. This is with MK8. Very frustrating as we are trying to start e-sports leaques.

[gregarican]

That's what happened when I opened my support ticket. The subject line was specifically "Firmware MX 18.208 Issues?" when I opened it. I was originally told that there were no known issues with this Stable RC. Apparently there are, as the ticket wound up confirming after specific testing and packet captures. Rolling it back brought our corporate network back to normalcy. It seems noticeable that Meraki was an acquired Cisco product line. Back in the day the old 1900 and 2500 series routers, ASA firewalls, etc. were pretty rock solid, granularly manageable, and stable firmware updates were pretty much the norm comparatively.