Z3 Teleworker VPN does not re-establish after outage

[aweise17]

After a power outage or an internet provider outage at my house, the Z3 teleworker appliance does not re-establish the VPN tunnel with our MX68 appliance in our data center. There seems to be no issue with the Z3 communicating with the Meraki dashboard, but the VPN tunnel does not re-establish.

I appeared to have an outage at my home overnight last night that recovered at approximately 1:15am EST. Since that time, I can see the MX68 continually attempting to communicate with the Z3 (via our firewall logs), but there seemed to be no response from the Z3 - thus, the VPN did not re-establish.

Are there logs on the Z3 or MX68 that I can view to determine why this happened?

[alemabrahao]

What version are you running?

Have you tried doing a factory default?

[aweise17]

I'm running MX18.107.2. I haven't tried a factory reset, but it's something I will look into.

[JChick]

I'm having the same problem with our Z3 Teleworker devices. Sometimes they work themselves out of it, sometimes they reconnect with no issues. It's very frustrating.

We have to fall back to having our users connect with the Cisco Secure Client, which defeats the whole purpose of deploying Z3's as a more secure method of connecting corporate equipment to our network.

I am on the latest firmware as of this date. MX 18.211.2

Upon reboot, my Z3s also connect to the Meraki cloud just fine, they even show that they have established a VPN tunnel with the MX68 but VPN status is down, and no traffic will pass through the tunnel.

[kchand]

Are you observing any packet loss on the WAN interface of the Z3 device at the time of the issue?

[JChick]

Yes,

I just updated the firmware on three of my Z3s yesterday and after reboot, they each took anywhere from 15 to 20 minutes before any traffic was routed down the WAN interface. (I probably should have documented it with some screenshots, but I just went about doing other tasks while waiting for them to come back).

So to clarify, upon reboot, within a few minutes they are detected by the Meraki dashboard, appliance status indicates an active wan connection, but if I initiate a ping to anything on the other side of the VPN, for 15 to 20 minutes I get 100% packet loss.

If I to to the Tools, I can ping the WAN appliance, run traceroute, MTR, DNS lookup, and ARP table all work fine, but I cannot ping anything on the other side or traceroute to the other side for 15 to 20 minutes.

If I navigate to the VPN status under the Monitor options it knows it is suppose to be looking for the MX68 peer, but the status dot is red and the connectivity indicator bar is red. Uplink decisions indicates that it is seeing the MX68, but no packets are flowing.

The three indicators: VPN Registry, NAT type and Encrypted are all green as well.

Hope that helps.

Very odd.

[Crocker]

Do you happen to have a bunch of Site-To-Site VPN Firewall rules? Or a small # of rules with a huge # of associated subnets/hosts?

We tried using Z3's in our network, and they worked pretty well for a year or two until we started building out VPN firewall rules. After some point, some critical mass, it was apparently too much for the Z3's to handle and they became very unstable after VPN connectivity blipped for any reason.

[JChick]

We do have a couple basic rules for the individual Z3s. Users are only permitted to use company devices with their teleworker appliances. The behavior I'm seeing seems more like a routing issue than Site-To-Site VPN rules. For example, I've noticed that even though we have IPV6 turned off internally, once the Z3s do sync up and start passing traffic, IPV6 routes show up in the routing tables as routes between the Z3 networks. I don't know if that has any bearing on anything, nor do I understand why they are established since we only use IPV4.