So it doesn't seem to matter, I know there are definitely issues with IKEv2 I would try and do IKEv1 if possible but even that is broken it seems. We have thrown everything at this problem and we have given up and are about to add an MX to the other side to hopefully resolve this ... View more

No news, I literally just had to bounce the Meraki side by making a change to the Site-to-site side of the tunnel which then brought all traffic selectors back up. I can do a similar thing to the FortiGate on our ASA side and bring the tunnel up as well. How many remote networks do you have on each side of your tunnel? I feel like we had some success in reducing the amount of networks configured on each side. ... View more

I need to ask you what you are using for Phase 1 and Phase 2 settings on both sides though if yours does stay up ... View more

Very sad news indeed, same issues have cropped up again. I swear it looks like the IKEv1 tunnel just can't handle more than 2 or 3 traffic selectors. This particular instance there are a ton, keep in mind these are all randomly generated addresses but will help with the example: MX Private Networks: 172.31.10.0/23 172.31.20.0/29 ASA Private Networks 10.100.100.0/24 10.9.102.0/24 123.88.139.8/32 123.88.139.19/32 123.88.139.25/32 123.88.139.92/32 148.66.118.201/32 148.66.116.78/32 148.66.118.202/32 148.66.116.79/32 148.66.99.74/32 10.5.131.36/32 10.5.131.37/32 172.22.26.0/24 The bolded ASA Private networks are the only ones we really care about and that are up most of the time. This effectively creates 6 (2 x 3) traffic selectors between the two devices. TS - 172.31.10.0/23 === 10.100.100.0/24 TS - 172.31.10.0/23 === 10.9.102.0/24 TS - 172.31.20.0/29 === 10.100.100.0/24 TS - 172.31.20.0/29 === 10.9.102.0/24 TS - 172.31.10.0/23 === 123.88.139.8/32 TS - 172.31.20.0/29 === 123.88.139.8/32 It looks like once it goes over 3 the next time a selector is re-keyed... its a no go. ... View more

I wonder if the outage yesterday and the fixes applied after reboot for MX running 18.211.x was the fix for all of the IPSec issues and that if I roll back to 18.211.2 if that will have fixed everything. https://community.meraki.com/t5/Security-SD-WAN/Issues-with-AutoVPN-incident/td-p/248249/jump-to/first-unread-message We are hesitantly about to purchase another MX to fix the issue but we really don't want to do this if we don't have to. ... View more

After 3 days of our own MX being on 19.1.3 one of the traffic selectors went down again and was not able to re-initate from the MX side. 😡 ... View more

I hope we can get an explantion for today's events to get over to our customers. ... View more

What is super concerning here is that Meraki took down our MXs to multiple customers causing outages to 30+ sites without as much as asking. This must be a very serious issue to be doing something this disruptive. Never seen anything this bad happen in awhile. I also find that its not Passthrough or Concentrator MXs that were affected. Appears to be random. And its not "Auto VPN" its "ALL" IPSec which was finally admitted.  Our environment was up and working until we were forcibly taken down when the fix was identified somewhere between 11 - Noon EST. Bad enough that it couldn't wait until off hours outside of 8:00 - 5:00. So I wish they would change the wording to be something other than "AutoVPN" outage. We have a lot of Non-Meraki IPSec tunnels that were taken down today. ... View more

Give it 3 or 4 days. When you have to reboot an MX it seems to stabilzie it for some time. When support initially rebooted our MX pair we were troubleshooting to disable some backend multicore support it was stable for about 4 days and then started to repeat the same behavior again.  ... View more

You are very brave for doing this. But thank you. I will be watching. As for the manual fix above. I am just talking about being able to bring up the non-working traffic selectors using packet-tracer from the ASA side in my case. ... View more

Anyway we can mark this as not solved? We confirmed how to manually fix this but we need this to be properly addressed by Meraki. ... View more

So first thing this morning, All of the traffic selectors survived through the night and are still up and incrementing encaps/decaps. ... View more

That's what support said in my case but they never actually confirmed or denied this. Feel free to reference my ticket number. They were unsure if the lower end (MX67) in my case were stable on 18.208 or 18.107 ... View more

Thought I would also throw out this other random mention of a bug that is not documented anywhere externally. Support is claiming there was a know issue with IKEv1 dropping random SAs: Case 12086900 Tunnels should not have taken that long to reform. Could be running into a known issue with IKEv1 connections on 18.107.10+ and 18.209+ firmware where we can't reform specific child sa's. Has anyone tested this theory by possible downgrading to any of the following? 18.107.8 18.170.9 18.208 18.208.01   All of these were considered "Stable" versions at some point. ... View more

So as I continue to watch the tunnel I see the VPN Registry: Partially connected warning. So I found another forum post here detailing what this means https://community.meraki.com/t5/Security-SD-WAN/VPN-Registry-Partially-connected-What-does-this-mean/m-p/36167   Appears that you can find what ports your registry is using with the following information and a Meraki Support Engineer telling us about having them change the registry values to fix issues:   From JosRus | Meraki Employee   I would like to add some additional information to this: When support initiates a change to your registry contact points, a migration period will occur, within which changes to your registry contact points cannot be performed. Any additional changes to these will require an additional waiting period while the migration finishes. An additional port of 9351 has been added, which you will see is also listed under Help>Firewall Info>VPN registry. Upon issuing a registry IP change from our side, you will see the addresses on this page update automatically, so be sure to check this page after any registry IP change is made from the Meraki Support side, and update your upstream firewall/device rules with the new information accordingly.     ... View more

So again first thing this morning one of our traffic selectors to a single host IP was not coming up from initiating on the MX side. I found this post for zScaler to MX and see them talking about recommended settings for Site-to-Site on the Meraki side: Follow these recommendations: Security & SD-WAN -> Configure: Site-to-site VPN -> Non Meraki VPN settings: Preshared secret must be greater than 14 characters Authentication cannot be MD5 Diffie-Hellman Group must be 14 Phase 2 encryption cannot be NULL PFS can be configured to be either off or 14   The only thing we are doing wrong may be that we thought the pre-shared key needed to be under 14 characters, probably another forum resource post somewhere.   I am not sure why they are recommendations and where its coming from other than others experiences I have scanned through the forum and have seen numerous posts now for similar issues that we are communicating about here: https://community.meraki.com/t5/Security-SD-WAN/NON-MERAKI-Site-To-Site-VPN-network-translation-v18-2xxx/td-p/246015 https://community.meraki.com/t5/Security-SD-WAN/Non-Meraki-VPN-IKEv2-issues/td-p/245734/ https://community.meraki.com/t5/Security-SD-WAN/MX-to-Cisco-FTD-Site-to-Site-Using-IKEv2/td-p/245108 https://community.meraki.com/t5/Security-SD-WAN/Cannot-establish-VPN-to-non-Meraki-peer-Firepower/td-p/240986 https://community.meraki.com/t5/Cloud-Security-SD-WAN-vMX/traffic-not-getting-initiated-from-IKEv1-for-non-meraki-tunnel/td-p/246867 https://community.meraki.com/t5/Security-SD-WAN/non-Meraki-VPN-peer-is-not-establishing-with-zScaler/m-p/182396 ... View more

Looks like we have the same topic going in multiple locations now, for the latest head on over to https://community.meraki.com/t5/Security-SD-WAN/Traffic-not-getting-initiated-from-IKEv1-and-IKEv2-for-non/m-p/246880#M55094MX VPN Drops for continued discussion on non-Meraki IKEv1 not initiating traffic over tunnels. ... View more

Thank you for confirming that without me having another false hope fix! Hopefully we get some visibility from this discussion so please keep checking in if you learn anything new.   Not excited because I manage a few very similar environments that so far have avoided this…. Which doesn’t help when trying to troubleshoot to compare the other setups as I have already tested and reviewed all of the configuration differences to no avail. ... View more

Looks like support actually answered a similar post from Tishman here about upgrading to the patch I mentioned 18.211.3 but again nothing in release notes to indicate a fix for what we are seeing:   https://community.meraki.com/t5/Cloud-Security-SD-WAN-vMX/traffic-not-getting-initiated-from-IKEv1-for-non-meraki-tunnel/td-p/246867 ... View more

I have been fighting this exact same issue between an MX (both 18.107.10 and 18.211.2) and ASA (running latest suggested code, 9.18 interim).   Support has both suggested trying the patch 18.211.3 to see if it fixes the issue and also saying they don't believe that will fix the issue.   I have changed about everything that I can on the ASA side including disabling keepalives and NAT-T. Adding 'route-lookup' to the end of the NAT exemption for the tunnel traffic Adding and subsequently removing the IKEv1 command 'reverse-route' Reordering ACL on ASA to match the order of private subnets on the Meraki. Its seems sometimes random when one of the traffic selectors will fail. But we can bring the specific failing selector up with this line in packet-tracer from the ASA side: packet-tracer input inside tcp [INSIDE SOURCE IP] [DESTINATION IP] 443 detail I have had two calls between TAC and Meraki Support and no solutions have been found. Meraki supports last action was to disable multi-core support on the backend for the MX67 in my case since it has been known to cause issues on the lower end devices. I thought this was going to fix the issue, it was stable for nearly 5 days and began to happen again... I think it was simply the reboot that fixed it which was necessary for the backend change. I am actively trying to get this case escalated on both supports sides but having no luck. Went straight to the Cisco Rep too... nothing as of yet but I will let anyone know if I fix this. I feel like it may have something todo with DPD / keepalives that at least one Meraki support engineer said on their end would be 'interval 10 retry 3' I belive the default on the ASA may be 'interval 10 retry 2'... But no telling what it is at this point. ... View more