How to reset single MX site-to-site VPN without rebooting?

[treimers]

Hi all -

I have an MX-450 that, among other functionality, supports multiple site-to-site VPNs to remote sites.

Yesterday, we had a fiber cut that took down one of the multiple sites, which of course took down the VPN to that site.

The other VPN site-to-site tunnels stayed up.

After fiber service was restored, that MX-67 at the remote site became available on the Meraki Cloud again.

But the VPN did not come back up, even after rebooting the remote MX-67W.

In Cisco ASA-land, this would be resolved by "clear crypto isakmp sa <tunnel group>" and the matching ipsec clear command. That would reset just the one tunnel on the host ASA side, and allow the VPN to restart.

I tried disabling/un-configuring the entire VPN config on the remote MX-67 - after 30 minutes, that hadn't done it.

In my case - the only way to restore the VPN was to reboot the host-side MX-450, which was highly disruptive

to many many more users--  including our entire police and fire department first responders in the field.

Not good, Meraki...

What is the equivalent command in the MX-450 to be less of a sledgehammer approach 

than rebooting the entire appliance as if it were consumer Linksys hardware?

Thanks Tim

to a problem affecting just one site? 

[Nash]

What kind of errors did you have in the log at the time the tunnel was down? Because that'll change my answer.

[treimers]

Mainly just vpn changes over the last few weeks as the remote sites lose power during thunderstorms --- but none of those messages correspond to the timeframe, except when I rebooted the MX.

The site where the fiber cut happened is a large one with full generator power.

[hoempf]

How exactly did you determine the state of the VPN? Usually IPSec devices (including Meraki) only establish a tunnel if there is traffic over the VPN, i.e. one packet from site A to site B is enough to trigger a reconnect.

I just wanted to rule out this one.

Did you call support btw? Sometimes there's suprisingly good 🙂

P.S. Just wondering: Who sold you just *one* MX450 if it's that important? The second MX does not need a license btw. As you described it, lives are at stake here 🙂

[treimers]

So, we do have a second MX -- that's what enabled me to reboot the primary unit without _serious_ problems,

although we did lose connectivity on public safety apps for a moment -- Not the end of the world, but not how I like to treat public safety either.... The applications recover, but it's still unprofessional to have no other recourse except to  reboot the entire  firewall to reset the IPSEC parameters on just one of many VPN ipsec tunnels.

I did talk to Support -- they made changes to the VPN registries available/known to the networks involved,

but that wasn't the problem - I didn't have the ability to be on the phone with Support from where I was at the time of the outage.

And - the other part of the question - Yes, there was traffic, the subnets full of Cisco phones and PCs on either end

definitely were still trying to talk - this is not a lack of "interesting traffic"

On an ASA, I'd have been able to do debug commands to see the ISAKMP attempted setups (or failures),

and capture match traffic on the interesting traffic ACLs, etc.

As  it was - all I could see was that the endpoints on both sides could not see each other.

Both MXes could see the MAC address of the other unit in ARP tables, but could not ping each other, but could ping other devices (gateway router, other public IP devices)  in the same public-side IP subnet.

Zero ability to see the tunnel "down" other than a red bar in the GUI ---  zero ability to see lifetimes, keepalives, etc.

That info should all be in the gui and monitorable in SolarWinds (or other SIEM/NMS).

So much more info exists about an IPSEC tunnel - none of it is visible in the GUI, and not in logs.

And no "button" to reset just one tunnel without dropping others.

[Nash]

You can actually do packet captures off Meraki devices in order to gather more information. Run it off the MX's WAN interface and you'll see p1 and p2 traffic, even on tunnels that are negotiated by AutoVPN. I just confirmed with a client that uses AutoVPN to connect multiple sites.

[treimers]

yeah, next time I'll do that.

Hopefully there won't be a next time, but still...

[MarcP]

Hm, we actually had the same problem, several times

All green on Meraki site, showing the VPN ist Up. But on ASA site it showed a failure. Data went into the tunnel but no response or anything else from Meraki site.

The ASA admin needed to reset the Tunnel all the time, so we didn't had to reboot the MX.

We weren't able to so anything. 

Since the tunnel is pointing to a fortigate it never happened again.

[treimers]

Hi Marc --

This is a Meraki to Meraki VPN.

My comments surrounding Cisco were from the "if I had the same commands in Meraki" perspective.

Not to confuse the issue and make people think one side was an ASA.

I've done hundreds of VPNs on Cisco ASAs...works great...But too expensive now

[GIdenJoe]

This is exactly my main beef with Meraki that when real problems occur you only have a green led and a limited log to troubleshoot and no real control over the inner workings.  Basically you have a steering wheel and a clutch.  With Cisco “Classic” you have a cockpit with loads of buttons and switches.

I wish they would have an expert mode on dashboard or a cli/API interpreter where you input commands or code and see direct results.

[Nash]

@GIdenJoeA significant design quality of Meraki is that the equipment is simplified, so the barrier of entry to using it is lower. Not having a trillion nerd knobs is kind of the point.

I sell it to a lot of people and very few of them need the loads of buttons and switches: they want equipment that they can manage via a GUI with fairly sensible design. Their networks are fairly simple and fit within Meraki's existing offerings. They want stuff that pretty much just works without them having to fiddle that much. The people who want or need nerd knobs buy... Cisco equipment. Because that's what Cisco is there for.

Would I like to be able to individually restart tunnels? Sure would!

But your argument is a bit like claiming newer cars are bad since a lot of work takes going to a mechanic. It's not like an old Jeep, where it's fairly easy to fix almost anything yourself. Many folks just need a car that pretty much runs fine with basic maintenance.

[GIdenJoe]

I understand your argument for the simplicity.

But would you want car manufacturers to take away the possibility to change your own tires if you have a flat somewhere on the road and you have to wait for a technician from them to get you back on the road?

That the config usually should be simple OK, that's true.  But for real troubleshooting you need an 'expert' mode.  Like when building VPN's to non-meraki peers it would be a great plus to actually see what's going on because a packet capture can't always see everytning (like the contents of the 3rd exchange in main mode IKE).

Cisco itself also introduces simplicity with DNA center but they still allow console access.  Just saying...

[TGTELVT]

Welcome to Meraki MX..

You can pair a ASA/Firepower, etc with the rest of the Meraki stack and get the better of both worlds..

then you get to vPC, etc and wish your nexus came back.

But for 97% of things Switching, Wireless and Video Meraki does it well with a little "Cloud Tax" on the speed.

[Looker44]

I also have this issue with a non Meraki peer. I don't see the answer of how to reset ONE VPN  in all these posts.

[Nash]

---

@Looker44 wrote:

I also have this issue with a non Meraki peer. I don't see the answer of how to reset ONE VPN  in all these posts.

---

Because you can't.

You can bounce tunnels two ways, basically. Both of these disrupt every tunnel you have up:

1. Disable site to site VPN. Save. Wait for config to take effect. Re-enable site to site VPNs. Save. Wait for config to take effect.

2. If you have multiple tunnels, rearrange their order on the list, save. It should tear em down and bring em back up.

[Looker44]

My initial response is "You are joking", but I have been in IT too long for that...

Since I have inherited this back-end, I assume i can just drag the Arrow icons up and down on the Non Meraki Peers sections.

THX

[Nash]

I'm sorry, @Looker44 but no, I'm not joking here. It's not my favorite thing in the world.

You do reorder by dragging the arrows up and down. Make sure you save after you do so.

[TheSimplifier]

Thanks for the tunnel bouncing procedure @Nash. I don't like it either, but it's just something we put into our P&Ps for now unless/until they add a feature to bounce individual tunnels. 

[pkinnison]

I understand the path Meraki is taking to "Simplify" the experience,   but to dumb down things to the point that only a reboot or a Support call is the only solution is not the answer.  

For items such as VPN tunnel creation and reset,   having 100's of VPN connections and not have the ability to just drop one VPN and reestablish is a serious shortcoming.       It is obvious shortcoming for an Enterprise solution.

Yes,  we have dual Meraki's in our Datacenters in Warm Spare configuration,   that isn't the point.  As end users we need the ability to do some of the basic management of the VPN tunnels without a reboot of an appliance failover for every remote VPN in the process.

Just my two cents,  and by the way.   If Meraki's is listening the ability to get ACCURATE information about he code level your Meraki is running would be helpful in the dashboard as well.   Right now it shows the level that you hope the device is on .   Many time have spoke with Meraki support to find out that firmware is stuck and not upgrading successfully.  Basic stuff guys.