Edit: I can see the full notes now and and am pasting. Original post still below for history of this breaking news 🙂
ORIGINAL POST:
Just got a ton of emails on a mystery 18.208 upgrade being pushed. Dashboard says:
The email says Significant performance improvements for MX75, MX85, MX95, MX105, MX250, and MX450 appliances.
My emphasis in bold, as many of us know the bolded models have not been announced as getting performance updates with this release. The rest of the email info seems to match what we do know about 18.2XX. A recent forum post says the other models are on the to-do list but no ETA. Anyway, I'm on hold with a confused support engineer and will update if he says anything useful, but wanted to let folks know.
It must be in the process of being published. I see the notes if I manually jump forward to that version on the RC release notes page.
And yes it appears to include the multicore improvements that came to the 75, 250, 450 in an earlier release.
Ok I found a shard where I can see that. That's fantastic news. Hopefully the exact numbers will be out shortly. Editing my main post to share.
My MX68CW is able to reach 1Gbps in firewall throughput :
What did you get with IPS/IDS on. Speedtest for my MX68 with those on I get 270Mbps on 18.107.8 upon upgrading to 18.207 or 18.208 speedtests dropped to 220Mbps. Seems slower
Honnestly I don't use the IPS/IDS module on the MX for many good reasons so I haven't tested yet. This is purely firewall throughput.
It is scary how hard this release is being pushed. It has only been out 24 hours or so, and I have zillions of upgrade emails. I don't think it should be rolled out so quickly.
18.207 was unstable for me. I am trying out 18.208 now, and after 4 hours or so it is good - but that is not enough testing time.
I don't know if this is accurate. The support rep didn't seem super knowledgeable about the release so I didn't previously mention, but he said something that seemed to indicate there may be an as yet to be announced security or similarly massive issue that required the quick deployment.
The update email talks about Detailed, live firewall logs can now be seen through a new live tool
A standard "feature" with other firewall manufacturers that I've been waiting for for a long time with Meraki. I'm really excited to see how the live view turns out and will therefore bring forward the update on one of our MX to tonight 😄
Does anyone else actually feel this way?
I used it when it first came out several months ago. It is great!
Wait, that's been out for a while? Apparently it completely passed me by...
I'm on 18.107.2
How long has this been out?
It came out with 18.207, the previous stable release candidate version
What kind of MX did you update? I have MX250 HA pairs, a 95 and a couple of 105s. Curious to see how it went.
It was a HA pair for 105s. This is what was being logged. There was no power issues, assumed the box was crashing. It's only passing SD-WAN traffic.
Feb 14 08:01:19 | Sterling DC - Primary | Device boot | Device boot | reason: power event or other |
I updated my HA MX95 last night and it looks like no issues so far. However, I didn't have any performance problems with the MX95 before.
Hi,
Installed 18.208 on MX85. Great performance increase.
But be warned we are experiencing issues with configured 1-to-1 NAT rules. MX dropping random packets as if there is something wrong in the session/translation table lookups
Good to know, thanks for the hint.
We are experiencing the same issues, I've narrowed it down to replies being blocked on stateful connections.
Interesting, this could have been the root cause of our MS Exchange issues where we saw extreme timeouts and disconnects.
Had the same problem with our 1 to 1 Nat rules. Had to revert back to 18.107.2.
Same here...
We also had the same issue and had to roll back our MX.
We also had to roll back to 18.107.2 on our MX250 HA pair. We were having issues with multiple devices that did have 1-to-1 NAT rules.
How does QA not test for 1:1 nat... Ill be doing manual updates from now on.
I wish I ran across this thread before updating to 18.208 this past Sunday. We have a couple of on-prem servers defined with 1:1 NAT on our MX75. They randomly lose outbound connectivity to external hosts/ports, which has caused major headaches with our apps. Most importantly, cloud data backups. Any other LAN host can connect to these endpoints without incident. It's only happening on two on-prem servers that have 1:1 NAT rules in place. Reverting the firmware back after hours tonight. If that does the trick then I'll be relieved!
After rolling back the firmware these two inside NAT 1:1 hosts are communicating outbound just fine. Previously it was spotty at best. I still am somewhat amazed that this firmware was considered a Stable Release Candidate. Even the most basic beta testing should've revealed NAT 1:1 hosts having connectivity issues. Reminds of this back in the day when I managed IT for a larger corporate call center --> https://rcpmag.com/articles/1999/11/16/microsoft-posts-winsock-hotfix-for-sp6.aspx.
I was just running into this issue as well, for the last week i have been having random outbound timeouts to external hosts from my internal 1:1 servers. I called meraki and they said oh yeah 18.208 is a release candidate. How would you push a RC to all customers. I am most defiantly frustrated with meraki right now.
did the roll back last night and my 1:1 servers are working again !
We've experienced the same problem and had to rollback to 18.107.x
Thanks for sharing
Installed 18.208 on a few MX85's with out issues, but the MX105 was a desaster, boxes were unstable and rebooting every 20 minutes or so. Rolled them back to 18.107.2
Hey @ToddB , do you mind sending me a DM with the serial number of the unit? I'd like to get this checked out a bit more.
Many thanks!
Giac
First time that 18.2xx was running completely smooth on MX85. Looking ver good, especially performance-wise so far!
Has anyone tried this on the MX250? I have two sites with MX250 HA pairs and may try it this weekend. I'm only able to push out the update a month, since getting the notice on 2/13. I'm not seeing an obvious compelling reason to rush the update, so I'm a little concerned. @ToddB I have a site with an MX105 as well. Was it rebooting itself or were you having to reboot?
I updated my HA MX95 last night and it looks like no issues so far. However, I didn't have any performance problems with the MX95 before.
What surprises me is the new firewall live view. Nothing happens when I want to look at some client traffic. Does anybody know what I'm doing wrong?
No matter which IP I take from different VLANS, nothing happens. The wheel spins for a moment and that's it.
Edit:
Oh man, I have a suspicion as to what it could be. We use client tracking through MAC addresses and I bet the live view only works with IP tracking. Unfortunately I can't use that because my MX95 are in a combined network, so I would have to split the networks first. Can someone confirm my suspicions?
However, there is no reference to this in the instructions.
https://documentation.meraki.com/MX/Firewall_and_Traffic_Shaping/Firewall_Logging
The live firewall logging is now broken on certain models running 18.208. It's not due to your client tracking mode. I discovered that night one and reported it to engineering.
Really? Fantastic Meraki, releasing a feature in one version only to kill it in the next..
Thanks anyway for reporting to support!
Well, it clearly wasn't an intentional move. It's been reported to engineering and I'm sure they'll resolve it.
a few weeks ago I tried 18.207, but is was very buggy and not working for me, because our servers in the DC are not reachable anymore (3rd party VPN between 2 MX's in different Orgs), switched back to 18.107.8 and everything works fine anymore. In 18.207 AND 18.208 the 3rd party VPN tunnel is shown as up in both locations, but no server is reachable via ping. MX18.208 has the same issues. I rolled back again to 18.107.8 and everything is fine again for me (MX85,MX95,MX100->VPN Concentrator MX450)
Hi holli69, just out of interest was you seeing this VPN issue on both the primary and secondary wan?
We upgraded a few weeks ago to the 18.207 and had no issues with the VPN untill a lorry broke the main fibre to the building, and the VPN was showing up on the secondary connection, but couldn't get it to send any data, downgraded.the firmware, and then it was back working.
HI tcanty,
at the moment we use VPN only at primary WAN Connection, secondary is in use for other traffic.
We couldn't send any data as well.
Hopefully in 18.209 or 18.210 it will be fixed.
We are also seeing this issue. We had 19 sites upgrade to 18.208 last night and this morning 5 of them had a problem routing VPN traffic over a Non Meraki VPN. The VPN Tunnel was shown as up both in the Meraki portal and the remote Cisco ASA device but no traffic was able to route over the tunnel. Devices affected were MX67s and one MX68CW-WW. Downgrade to 18.107.2 on the MX67s and 18.107.8 on the MX68CW-WW has seemingly fixed the issue for now.
Anyone using 18.208 on the MX67C? Still stuck on 17.10.9 because of all the issues with integrated cellular in every subsequent release.
we have 18.207 running on most of our mx67c's, and other then the vpn thing mentioned just above, we haven't had issues with the cellular, and 208 is meant to have further improvements on that side. We aren't going to be rolling to 208 due to the issues with the VPN, and are slowing moving back to the 18.107 versions, however that then breaks the cellular part.
Another observation: Device Utilization went down noticeable:
We also switched back to 18.107, because we have some problems with our 3rd party Zscaler tunnels, which seems to make trouble after 1-2 days.
@Fabian1 , do you have any more details on what kind of problems you experienced? Do you have a case with Support?
Many thanks!
Giac
No case
We have a 3rd party VPN tunnel to Zscaler with a default route 0.0.0.0/0 and after a short time the tunnel was still up on dashboard but no traffic went over the tunnel
We have also routes to MPLS (LAN) and a AutoVPN to AWS/vMX, both were fine
With the stable, we do not having these issues
We haven't noticed any issues with either 18.207 or 18.208. We've been running 18.207 for about a month, and I've deployed 18.208 to a handful of MX68 devices shortly after the update became available. Pushing out 18.208 to all but one of our MX68 and MX250 devices tonight. Our configurations though don't include what others have mentioned here as issues.
Are you running your MX250s as HA pairs? Using Meraki VPN?
Just curious because I have MX250s and will be upgrading over the next couple of weeks.
The MX250s are not currently running in HA. We are using Meraki VPN though.
Thanks, I appreciate the info. I hope the updates are successful tonight.
FYI, this version is listed as a "stable release candidate", not "stable". We upgraded our HA paired MX250s early this morning and have had multiple VPN outages and our Master has failed over to the Spare 4 times since 7:30am. I am scheduling a rollback this evening. There is also no way to opt-out of firmware upgrades that are anything but "stable". This has to be a manual selection to reschedule or cancel.
Thanks for the info.
I've found that you can reschedule the update for up to 30 days.
I received the first notification on 2/13 and rescheduled for 3/13.
I went back on 2/16 and was able to reschedule for Saturday 3/16, so it looks like a sliding 30 day window.
I'm just going to keep moving the window until it is safe to install.
You can also cancel it completely, rather than keep moving it.
Didn't see that option until you mentioned it.
You have to go to the 'Scheduled changes' screen, then you have it there.
I was on the 'Schedule upgrades' screen and it only have upgrade now or reschedule.
Thanks.
I have one user that after the upgrade to Meraki MX 18.208 (MX250, just last night) won't connect to VPN any longer. Had 2 calls with Meraki and the VPN connection won't most often even ask for credentials and based on packet capture fails at step 2 of negotiation. We were able to connect 2 times successfully, but most attempts fail.
Client is on latest Windows 11 (23H2) (updated after it was already not working, this morning)
Meraki MX250 on its latest MX 18.208
Recreated VPN IPSec connection (do not have AnyConnect licenses).
Other VPN users are connecting just fine which points to a client issue but thought I would bring it up here for some resolution?
In the past, this was a MS update that needed to be uninstalled for VPN to work again. Any ideas?
We seem to be having the same issue. I'm getting ready to contact support to downgrade back to our previous version that didn't have any VPN issues. Have you found anything else out about this?
We have downgraded all our mx's back to the older firmware, and the function returns to working. They have updated the firmware notes to say there is an issue with none meraki VPN over a ppoe connection, so seems to be known by them. You shouldn't necessarily need to contact support to downgrade, if you go to the firmware page it lets you "upgrade" to the latest stable version, once selected it tells you are downgrading.
Can can schedule the rollback your self still if your within 2 weeks, Network-> Organization -> Firmware Upgrades - > Click the rewind arrow for the upgrade. Fill out the information and it gives you a window to schedule the roll back, i did mine last night back to 18.107.2 and my 1:1 nat issues are fixed now.
There really needs to be a opt out of RC and beta builds.
I was able to cancel the upgrade. I was hesitant to apply it until I saw other people using it, so I scheduled it 30 days out. At first I thought all I could do was move the date, but figured out I could cancel it if I went to 'Organization' -> 'Firmware Upgrades' - 'Scheduled changes'. At that screen you will see the scheduled upgrade and will have the option to cancel. Once I did that, it was not pushed again.
I definitely agree they shouldn't be pushing an RC like this in the first place. They don't make it easy or intuitive to cancel. It would have been much better to have a 'Skip this upgrade' option or something in the first notice. If I want to be a test bed for Meraki, I'll go get the software myself. I don't need it done automatically. That's not a stable way to manage infrastructure.
Yeah i think this will be me go to action now, especially since it seems we cant trust meraki to QA their builds or to not give us a release candidate.
18.208 was a disaster for us. Pushed it out last night, came in today to many issues. Specifically, on-prem MS Exchange server had excessive timeouts and interrupted sessions. It would log ASP errors in the event log, and from the client side, OWA would frequently not work, time out, or give random errors. Outlook full application would time out/disconnect. Also noticed websites would time out, then eventually reconnect several times. For example, was downloading a script from GitHub and the webpage timed out, then reloaded, then downloaded 2-3 copies of the file we were trying to download. These were all on MX105s and 95s.
Also had issues with an MX75 after the 18.208 update where VOIP traffic was not working. Rolled back, issues gone.
I would NOT recommend this firmware at this time unless you are very closely monitoring everything - these are just the issues we found briefly using it.
I can confirm that 18.208 broke things for us this morning. We have x2 MX250s in HA and MS250 HA switches behind it. After the upgrade one of the switches uplink shows disconnected and switch became unreachable. We have downgraded back to 18.107.2 and it all started working OK.
Our upgrade was scheduled over the weekend and we had to roll back to 18.107.2 this morning on our MX250 HA pair. SIP traffic was giving us one way audio if the SIP gateway would register at all. Office 365 connectivity issues, and Cisco Duo connectivity issues.
We have an HA pair of MX95's that are having issues with split tunnel in Any Connect. We pointed our users to an MX67 with the same firmware an it works fine so I don't know if the issue is specific to the MX95, or the HA setup.
What a nightmare. MX250 in HA keeps failing over and all kinds of madness ensues. We just took one of the MX250 offline completely to stop the failover from happening. Support acted like everything was fine, it's not.
Another issue identified, our esports team reported that none of their Nintendo Switches would connect for their scheduled NJCAA match, and they may have to forfeit if they can't get it rescheduled. The error was NAT related.