15.42.1 and 15.42 latest OS versions breaks routing and makes Client VPN unstable

[nsingh]

I am specifically posting my experience with Meraki Support and Meraki latest "Stable" release 15.42.1 

I upgraded our "X network" MX box on this sunday at 4.30pm PST.

To start with, it's been a very poor code quality since Meraki released 15.42 and 15.42.1.

Taking some packet captures, revealed that there is connectivity and routing issues between Site-to-Site Meraki peers, so while on one location client VPN, you cannot access the resources of the another location, there are intermittent packet loss.

Since the same day we started experiencing the connectivity issues in our Client VPN. These are the behaviours.

1). I connect to our "X network" client VPN on Meraki. Connects fine. Then within one minute the VPN disconnects automatically with no error message nothing.

2). I connect to our "X network" client VPN. Connects fine. No internet works on the VPN, internal and external. I can see packets going out, but no return traffic.

3). I connect to our "X network" client VPN. Connects fine. Internal traffic does not work, external traffic works.

4). I try to connect to our "X network" client VPN. It gives me Authentication failed for the same exact credentials that are saved in my VPN profile which was previously authenticating without any issues.

I have downgraded the "X network" MX box on 14.53, and now everything works fine. This happened on both 15.42 and 15.42.1.

Here is a sneak peak of other blogs of users facing similar issues - https://www.reddit.com/r/meraki/comments/n5hygl/mx_15421_breaks_routing_somehow/

Last but not the least - I have been on support hotline for about 40 Mins now, but no takers of my call. Kudos!

[CharlesIsWorkin]

So. back to 14.53 was the fix huh? Bummer that stuff is broke with the new supposedly "stable" update. I cancelled my scheduled update after reading this and other posts.

[nsingh]

@CharlesIsWorkin  - One of the community member gave me a suggestion to upgrade the downlink Meraki switches to the latest beta version 14.21 which seems to have reduced some issues for my team, however, we are still testing the 15.42.1 on one network.

[PDSKturley]

Same issue here Site to Site VPN will not negotiate on NW firmware! have an active case! Escalated to development several days ago.

Case:
06504065

Issue: Non-Meraki VPN tunnels will be down after MX100 is upgraded to 15.42.1

Actions:
+Data center MX100: Q2JN-XXXX-XXXX
+MX100 is upgraded and now non-meraki tunnels with Z1s and Z3s would be down
+Q2TN-XXXX-XXXX (Myrtle)Z3 with new firmware will work whereas all others Zs will not connect on new firmware, for example Q2TN-XXXX-XXXX(Tony Beach)
+Q2KY-XXXX-XXXX is another MX68 acting as a data center will have the same issue with tunnels after the upgrade
+All Zs will show phase one as up but phase two would be down. MX100 will not have phase one 1 up
+Took pcaps and collected logs for working and non-working condition.
+Rolled back the firmware and everything came back online
+Reach out to the internal team with the collected info

Next action due:
+Reach out to the internal team with the collected info

[Bruce]

@PDSKturley there have been a couple of threads here with non-Meraki site-to-site VPNs and one of the resolutions is to ensure the RemoteID and LocalID on the peer configuration are correctly configured. It appears more rigorous checking of identity was implemented between MX14 and MX15 firmware, and although you could get away without this on MX14, identity checks fail on MX15 and Phase 2 fails. You don’t mention which firmware version you were upgrading from.

It doesn’t ‘feel’ like this is the answer to all the issues you highlighted, but might be the answer to some of them.

[PDSKturley]

Thanks for that insight. More details for those reading this thread as I spent a couple hours on with a knowledgeable Meraki tech, but the only resolution so far was to Roll Back the MAIN (Data center) MX68 and MX100.

Edge routers, a mix of MX64, Z1, and Z3 devices. (All Non-Meraki VPN)

3 weeks ago we updated all the edge (Branch routers) to whatever the recommend latest firmware was, Some went to 15.42.1, others were older Z1 and went to 14.53.

Next day after update we get calls from branches that their down, look at dashboard and VPN shows up. But we cannot pass traffic over the VPN. Reboot Branch router (the one with New FW) and VPN reconnects and data passes.

Works all day then another site calls same issue, same fix. 

We assume it is because the Datacenter Routers (MX68 and MX100) are not updated, however we don't have a maintenance window for 2 more weeks to update. We wait, and just reboot the problematic branches early in the morning to mitigate the calls. Problem persists and we roll back a few to 14.53 and hey become stable for the next two weeks. 

Maintenance day comes and we update all branch routers we had rolled back and the Datacenter routers.

It has now been just past the 2 week roll back for some branch routers we were not getting as frequent calls so they were not touched.

After Maintenance was completed the techs started testing connectivity to branches, all were down and would not come online, restarted all branch routers and 1 Z3 and an MX64 connected, but none of the other 20 locations would connect Phase2. Rolled back an MX64 and still nothing.

Called Meraki Support, spoke to tech who did lots of packet capture to determine in fact it was Phase 2 not negotiating. Tried RemoteID, still no connection. Tried roll back one branch, no connection.

Rolled back Data center MX100 - -  all reconnected!

I would love to know more about the RemoteID and LocalID setup if anyone can expand on that if it resolve tehis similar issue in your environment.

 Meraki support requested another call,  but it requires all the branches be offline during testing, so it mat be a few days before the client (who likes to work 24X7) will let me take it down at a legitimate hour so I don't have to be up from midnight to 4 AM troubleshooting 😞

[tps_koulnick]

I upgraded my MX from v14 to v15.

The RemoteID/LocalID where blank.

I have 2 Sonicwall peers.

One would send traffic only 1 way.

The other would fail on phase 2.

As soon as I added RemoteID/LocalID, both Sonicwalls connected immediatly.

THANKS!!!

[NetworkFeller]

What did you use for the Remote/LocalID?

[nsingh]

The issue is with Client VPN, not the Site to Site. Thanks!