MX68 Site to Site VPN - Juniper SSG Series - Drops

[Joe-Phillips]

Hi Everyone,

I am wondering if anyone here can point me in the right direction as this is driving me absolutely insane. I have had a case open with Meraki Support for over a week, but unfortunately I am getting nowhere fast.

We have a Meraki MX68 firewall which we have recently put in place of our end of life Juniper SSG5. We have two Juniper SSG520's in each of our datacentres which we have a site to site VPN configured to each firewall with multiple remote subnets. Historically when using a site to site VPN from our Juniper SSG5 to both Juniper SSG520's we had no problems at all.

Since configuring the Meraki we have had many issues with these VPNs dropping constantly. Initially they were going red on the Meraki dashboard but some of the remote subnets associated to that VPN would still echo ICMP but some would not. After many configuration changes we are now at a point where it seems the VPNs are stable but then drop after the 28800 life time has expired. The Meraki VPN status goes red but some remote subnets re-establish and echo ICMP, others do not.

I have checked the phase 1 proposal and phase 2 proposal so many times on both local and remote firewalls. They definitely match, as do the remote subnets configured on the MX68 and SSG520's.

Phase 1: PRE-G2-AES256-SHA1-28800

Phase 2: NOPFS-G2-AES256-SHA1-28800

These four events constantly appear in the Meraki event log when the 28800 life time has expired

Jan 16 17:14:15		Non-Meraki / Client VPN negotiation	msg: phase1 negotiation failed due to time up. c740a9f647b344d6:d607686ae6596e8c
Jan 16 17:14:13		Non-Meraki / Client VPN negotiation	msg: request for establishing IPsec-SA was queued due to no phase1 found.
Jan 16 17:14:08		Non-Meraki / Client VPN negotiation	msg: IPsec-SA expired: ESP/Tunnel MERAKIIP[500]->JUNIPERIP[500]
Jan 16 17:14:03		Non-Meraki / Client VPN negotiation	msg: initiate new phase 1 negotiation: MERAKIIP[500]<=>JUNIPERIP[500]

I see these events on the Juniper SSG520

2019-01-16 17:14:50	info	IKE MERAKIIP Phase 1: Retransmission limit has been reached.
2019-01-16 17:14:02	info	IKE MERAKIIP Phase 1: IKE responder has detected NAT in front of the remote device.
2019-01-16 17:14:02	info	IKE MERAKIIP Phase 1: IKE responder has detected NAT in front of the local device.
2019-01-16 17:14:02	info	IKE MERAKIIP Phase 1: Responder starts MAIN mode negotiations.

If anyone can point me in the right direction I would be most grateful.

Thanks in advance.

Joe

[Cmiller]

I had a very similar issue at a connection we built to a hosted data center. I was working for an old company and I'm going though my notes but I we had them disable one feature on the Juniper and life was grand after, just can't remember what feature that was. I knew i would need to know this again one day so I save the information, just have to remember where...

[PhilipDAth]

Your Phase 1 and 2 crypto settings must be right, because the VPN is operating for some period of time. You could ptentially check the times match at both ends.

It sounds like the VPN re-negotiation is failing.  This should happen before the original VPN expires.  It may be that the two ends handle this in a different way - and may not be resolvable (at least not without firmware upgrades).

On the Meraki side, I would tend to use the 14.x firmware, such as 14.37.

I also see that you are running the VPN with NAT between the two ends.  You should avoid this kind of configuration.  Are you able to get public IP addresses on the two VPN devices directly (the MX and the Juniper)?

[Joe-Phillips]

Thanks for the reply Philip. 

Yes, the crypto settings and timings definitely match on both sides. I have checked many times and had this confirmed by the Meraki engineer working on the case. 

The appliance is current running firmware version 14.32. It says this is latest firmware? 

With regards to your last paragraph, would you mind elaborating as this part I am unsure on. NAT-T is enabled, but I am unsure what the implications are of turning this off. All firewalls in this setup are configured with public addresses. 

Thanks

joe

[Cmiller · Accepted solution]

NAT traversal... we had them disable it on the Juniper and traffic flowed great. Try and see if it works magic for you

[Joe-Phillips]

Thanks for the reply! 

I have read a lot about NAT Traversal causing issues but I’m not 100% sure what the implications will be if it’s turned off. I can turn it off on the Junipers as I have control over these, although I have read that only Meraki support can disable NAT-T on the MX appliance. 

I kind of wish we had had stuck with local Junipers, this has been a headache for the best part of four weeks! 

[Cmiller]

Sadly there is only one way to test this..... and I wish you luck. I can say we had 25 VPN tunnels all with the same issue you are describing. The EMR software we used would kick a user out after 1... yes 1 dropped packet. So we were desperate to get something working. Nat T fixed it for that client.... Maybe a quick test after hours with a glass of whiskey and maybe it will work for you too 😉 

[Joe-Phillips]

This is true. I’ll speak with our solutions architect tomorrow and see I can turn it off after hours and see what happens! I have a feeling this may solve the issue as I found similar threads on PFSense and Sophos communities, all of which were using tunnels to a Meraki appliance.

Do you know if you just disabled NAT-T on the Juniper firewall or did you have Meraki disable it on the MX appliance too?

I’ve been so impressed with the MX from an overall perspective, but I have to say the lack of VPN features (dpd, heartbeat, P2 SHA256, etc) is disappointing as is the event log. The logs I saw on Monday were something like ‘Hmmmmmm...’. I thought that was my job to say that, not the firewall!

[Joe-Phillips]

So I came in to the office this morning and surprise surprise one of the VPNs was down. I decided to take the plunge and disabled NAT-T on the Juniper firewalls and the VPN immediately came up. They've been active since along with minimal log entries on the Meraki side which is good. I'm still not 100% sure what the implications are of disabling NAT-T (I would like to know if anybody can explain). I've done some research but at the moment we are not seeing any abnormal behavior when communicating with the remote subnets.

I guess I will have to wait for 17:23 today when the 28800 lifetime expires and see if the rekey process runs smoothly before hand. If it does, I think we're on to a winner!

Thanks again for both of your inputs/replies on this. Here's hoping disabling NAT-T is the solution. I shall report back later on this evening. 

Joe

[Joe-Phillips]

Hey Guys,

Just to follow up from my message yesterday. At face value everything seems to be working OK and we haven't had any drops that I have seen. So I am really hoping this has resolved our problems. I did however see these logs from the Meraki MX68 appliance which resemble the issue I was seeing originally with P1. Any ideas?

2019-01-18T04:31:47.508304+00:00 192.168.171.254  1547785907.496919622 TTS_MX68 events Site-to-site VPN: initiate new phase 1 negotiation: MERAKIEXTIP[500]<=>DC1EXTERNALIP[500]

2019-01-18T04:31:47.516020+00:00 192.168.171.254  1547785907.504887982 TTS_MX68 events Site-to-site VPN: ISAKMP-SA established MERAKIEXTIP[500]-DC1EXTERNALIP[500] spi:3cd860a02087e03a:ba6b1e5740fe5fd6

2019-01-18T04:31:47.519521+00:00 192.168.171.254  1547785907.508372222 TTS_MX68 events Site-to-site VPN: IPsec-SA expired: ESP/Tunnel DC1EXTERNALIP[500]->MERAKIEXTIP[500] spi=111310359(0x6a27617)

2019-01-18T04:31:47.522957+00:00 192.168.171.254  1547785907.511800582 TTS_MX68 events Site-to-site VPN: initiate new phase 2 negotiation: MERAKIEXTIP[500]<=>DC1EXTERNALIP[500]

2019-01-18T04:31:47.526451+00:00 192.168.171.254  1547785907.515307462 TTS_MX68 events Site-to-site VPN: IPsec-SA expired: ESP/Tunnel MERAKIEXTIP[500]->DC1EXTERNALIP[500] spi=161914601(0x9a69ee9)

2019-01-18T04:31:47.532091+00:00 192.168.171.254  1547785907.520900662 TTS_MX68 events Site-to-site VPN: IPsec-SA established: ESP/Tunnel MERAKIEXTIP[500]->DC1EXTERNALIP[500] spi=160060028(0x98a527c)

2019-01-18T04:31:47.535577+00:00 192.168.171.254  1547785907.524382942 TTS_MX68 events Site-to-site VPN: IPsec-SA established: ESP/Tunnel MERAKIEXTIP[500]->DC1EXTERNALIP[500] spi=161914625(0x9a69f01)

2019-01-18T04:32:27.686331+00:00 192.168.171.254  1547785947.674877462 TTS_MX68 events Site-to-site VPN: ISAKMP-SA deleted MERAKIEXTIP[500]-DC1EXTERNALIP[500] spi:fa669a457fa376a1:bc2d0224f2fc98c8

2019-01-18T04:35:22.364064+00:00 192.168.171.254  1547786122.351001222 TTS_MX68 events Site-to-site VPN: IPsec-SA expired: ESP/Tunnel DC1EXTERNALIP[500]->MERAKIEXTIP[500] spi=241422734(0xe63d18e)

2019-01-18T04:35:22.367380+00:00 192.168.171.254  1547786122.354466942 TTS_MX68 events Site-to-site VPN: initiate new phase 2 negotiation: MERAKIEXTIP[500]<=>DC1EXTERNALIP[500]

2019-01-18T04:35:22.371087+00:00 192.168.171.254  1547786122.357958622 TTS_MX68 events Site-to-site VPN: IPsec-SA expired: ESP/Tunnel MERAKIEXTIP[500]->DC1EXTERNALIP[500] spi=161914602(0x9a69eea)

2019-01-18T04:35:22.376667+00:00 192.168.171.254  1547786122.363647142 TTS_MX68 events Site-to-site VPN: IPsec-SA established: ESP/Tunnel MERAKIEXTIP[500]->DC1EXTERNALIP[500] spi=108457053(0x676ec5d)

2019-01-18T04:35:22.380139+00:00 192.168.171.254  1547786122.367131942 TTS_MX68 events Site-to-site VPN: IPsec-SA established: ESP/Tunnel MERAKIEXTIP[500]->DC1EXTERNALIP[500] spi=161914628(0x9a69f04)

2019-01-18T04:35:24.388957+00:00 192.168.171.254  1547786124.375866222 TTS_MX68 events Site-to-site VPN: IPsec-SA request for DC1EXTERNALIP queued due to no phase1 found.

2019-01-18T04:35:24.392323+00:00 192.168.171.254  1547786124.379299302 TTS_MX68 events Site-to-site VPN: initiate new phase 1 negotiation: MERAKIEXTIP[500]<=>DC1EXTERNALIP[500]

2019-01-18T04:35:24.489306+00:00 192.168.171.254  1547786124.476226902 TTS_MX68 events Site-to-site VPN: IPsec-SA expired: ESP/Tunnel MERAKIEXTIP[500]->DC1EXTERNALIP[500]

2019-01-18T04:35:24.538770+00:00 192.168.171.254  1547786124.525605102 TTS_MX68 events Site-to-site VPN: initiate new phase 1 negotiation: MERAKIEXTIP[500]<=>DC1EXTERNALIP[500]

2019-01-18T04:35:38.428056+00:00 192.168.171.254  1547786138.414931262 TTS_MX68 events Site-to-site VPN: IPsec-SA expired: ESP/Tunnel DC1EXTERNALIP[500]->MERAKIEXTIP[500] spi=128968108(0x7afe5ac)

2019-01-18T04:35:38.431514+00:00 192.168.171.254  1547786138.418407342 TTS_MX68 events Site-to-site VPN: initiate new phase 2 negotiation: MERAKIEXTIP[500]<=>DC1EXTERNALIP[500]

2019-01-18T04:35:38.435044+00:00 192.168.171.254  1547786138.421983262 TTS_MX68 events Site-to-site VPN: IPsec-SA expired: ESP/Tunnel MERAKIEXTIP[500]->DC1EXTERNALIP[500] spi=161914603(0x9a69eeb)

2019-01-18T04:35:38.440864+00:00 192.168.171.254  1547786138.427813822 TTS_MX68 events Site-to-site VPN: IPsec-SA established: ESP/Tunnel MERAKIEXTIP[500]->DC1EXTERNALIP[500] spi=23444922(0x165bdba)

2019-01-18T04:35:38.444521+00:00 192.168.171.254  1547786138.431317902 TTS_MX68 events Site-to-site VPN: IPsec-SA established: ESP/Tunnel MERAKIEXTIP[500]->DC1EXTERNALIP[500] spi=161914629(0x9a69f05)

2019-01-18T05:19:52.011480+00:00 192.168.171.254  1547788792.007805188 TTS_MX68 events Site-to-site VPN: IPsec-SA request for DC1EXTERNALIP queued due to no phase1 found.

2019-01-18T05:19:52.014959+00:00 192.168.171.254  1547788792.011281548 TTS_MX68 events Site-to-site VPN: initiate new phase 1 negotiation: MERAKIEXTIP[500]<=>DC1EXTERNALIP[500]

2019-01-18T05:19:52.114711+00:00 192.168.171.254  1547788792.110962988 TTS_MX68 events Site-to-site VPN: IPsec-SA expired: ESP/Tunnel MERAKIEXTIP[500]->DC1EXTERNALIP[500]

2019-01-18T05:19:52.163888+00:00 192.168.171.254  1547788792.160150908 TTS_MX68 events Site-to-site VPN: initiate new phase 1 negotiation: MERAKIEXTIP[500]<=>DC1EXTERNALIP[500]

2019-01-18T06:05:24.898539+00:00 192.168.171.254  1547791524.895722717 TTS_MX68 events Site-to-site VPN: IPsec-SA request for DC1EXTERNALIP queued due to no phase1 found.

2019-01-18T06:05:24.901917+00:00 192.168.171.254  1547791524.899156917 TTS_MX68 events Site-to-site VPN: initiate new phase 1 negotiation: MERAKIEXTIP[500]<=>DC1EXTERNALIP[500]

2019-01-18T06:05:25.001729+00:00 192.168.171.254  1547791524.999007677 TTS_MX68 events Site-to-site VPN: IPsec-SA expired: ESP/Tunnel MERAKIEXTIP[500]->DC1EXTERNALIP[500]

2019-01-18T06:05:25.051326+00:00 192.168.171.254  1547791525.048469797 TTS_MX68 events Site-to-site VPN: initiate new phase 1 negotiation: MERAKIEXTIP[500]<=>DC1EXTERNALIP[500]

2019-01-18T06:11:22.308806+00:00 192.168.171.254  1547791882.302748770 TTS_MX68 events Site-to-site VPN: IPsec-SA expired: ESP/Tunnel DC1EXTERNALIP[500]->MERAKIEXTIP[500] spi=241422734(0xe63d18e)

2019-01-18T06:11:22.312095+00:00 192.168.171.254  1547791882.306216050 TTS_MX68 events Site-to-site VPN: IPsec-SA expired: ESP/Tunnel MERAKIEXTIP[500]->DC1EXTERNALIP[500] spi=161914602(0x9a69eea)

2019-01-18T06:11:38.372945+00:00 192.168.171.254  1547791898.366777994 TTS_MX68 events Site-to-site VPN: IPsec-SA expired: ESP/Tunnel DC1EXTERNALIP[500]->MERAKIEXTIP[500] spi=128968108(0x7afe5ac)

2019-01-18T06:11:38.376861+00:00 192.168.171.254  1547791898.370735034 TTS_MX68 events Site-to-site VPN: IPsec-SA expired: ESP/Tunnel MERAKIEXTIP[500]->DC1EXTERNALIP[500] spi=161914603(0x9a69eeb)

2019-01-18T06:49:52.534195+00:00 192.168.171.254  1547794192.525419519 TTS_MX68 events Site-to-site VPN: IPsec-SA request for DC1EXTERNALIP queued due to no phase1 found.

2019-01-18T06:49:52.537742+00:00 192.168.171.254  1547794192.529003599 TTS_MX68 events Site-to-site VPN: initiate new phase 1 negotiation: MERAKIEXTIP[500]<=>DC1EXTERNALIP[500]

2019-01-18T06:49:52.642858+00:00 192.168.171.254  1547794192.634054559 TTS_MX68 events Site-to-site VPN: IPsec-SA expired: ESP/Tunnel MERAKIEXTIP[500]->DC1EXTERNALIP[500]

2019-01-18T06:49:52.692211+00:00 192.168.171.254  1547794192.683358439 TTS_MX68 events Site-to-site VPN: initiate new phase 1 negotiation: MERAKIEXTIP[500]<=>DC1EXTERNALIP[500]

2019-01-18T07:35:25.422636+00:00 192.168.171.254  1547796925.414192135 TTS_MX68 events Site-to-site VPN: IPsec-SA request for DC1EXTERNALIP queued due to no phase1 found.

2019-01-18T07:35:25.427873+00:00 192.168.171.254  1547796925.419509855 TTS_MX68 events Site-to-site VPN: initiate new phase 1 negotiation: MERAKIEXTIP[500]<=>DC1EXTERNALIP[500]

2019-01-18T07:35:25.524170+00:00 192.168.171.254  1547796925.515681175 TTS_MX68 events Site-to-site VPN: IPsec-SA expired: ESP/Tunnel MERAKIEXTIP[500]->DC1EXTERNALIP[500]

2019-01-18T07:35:25.575097+00:00 192.168.171.254  1547796925.566681855 TTS_MX68 events Site-to-site VPN: initiate new phase 1 negotiation: MERAKIEXTIP[500]<=>DC1EXTERNALIP[500]

2019-01-18T08:19:53.055303+00:00 192.168.171.254  1547799593.040411685 TTS_MX68 events Site-to-site VPN: IPsec-SA request for DC1EXTERNALIP queued due to no phase1 found.

2019-01-18T08:19:53.058657+00:00 192.168.171.254  1547799593.043940525 TTS_MX68 events Site-to-site VPN: initiate new phase 1 negotiation: MERAKIEXTIP[500]<=>DC1EXTERNALIP[500]

2019-01-18T08:19:53.157632+00:00 192.168.171.254  1547799593.142691805 TTS_MX68 events Site-to-site VPN: IPsec-SA expired: ESP/Tunnel MERAKIEXTIP[500]->DC1EXTERNALIP[500]

2019-01-18T08:19:53.208838+00:00 192.168.171.254  1547799593.193982445 TTS_MX68 events Site-to-site VPN: initiate new phase 1 negotiation: MERAKIEXTIP[500]<=>DC1EXTERNALIP[500]