Issues with AutoVPN (incident)

[SamerAl]

Hello community,

We'd like to inform you of an ongoing incident affecting AutVPN it is affecting passthrough MXs on 18.2.x, We are keeping status.meraki.com updated as we continue our remediation work.

If you are experiencing this issue, please try to reboot the MX as this might help resolve the issue.

Best Regards 

Samer 

[DarrenOC]

Hi @SamerAl , for clarity, is that reboot all MXs involved in the auto-VPN or would a reboot of the main Concentrator resolve the issue?

[Shubh3738]

After Reboot the VMx Services started.

But after the sometime issue is happened again.

[SamerAl]

@Shubh3738 did you make any changes after the reboot( added a hub/spoke ) ?

if this is the case it might break again, our Team is still working on a permanent fix for the issue, so please keep an eye on status.meraki.com , if your issue is urgent and massively impacting the business please try to get in touch with our support team.

You can find our 24/7 support number when you go to the dashboard? > Get help and cases  > Pick a Tile to Contact Support > Call Support team

[SamerAl]

@DarrenOC , I would recommend rebooting the Concentrator first as the issue is affecting MX's in passthrough.

[RaymondH]

Rebooting only the Hub MX resolved the issue for us, at least for now. Hope it lasts..

[martin-netx]

We found that a restart of the vMX only was enough to bring back the VPNs.

[Shubh3738]

Iam not making any changes.

Just waiting for the confirmation from Meraki side to resolve that issue permanently.

[SamerAl]

The team Identified the issue and they are working hard to roll out a permenant fix, I will update this thread when the issue is resolved.

[finzwake]

Tunnels dropped specifically to our VPN concentrator Hub (running 18.211.2) around 1:06am EST. MXs in routed mode don’t appear affected.  Tunnels appeared up on VPN Status page for all MXs.

Rebooting the Hub concentrator restored tunnels, for now

[RaymondH]

About same experience, tunnels seemed to be up, tracert worked, but no dns, ping or Umbrella connection on the spoke MX's.

[Felix_moreno]

2 out of my 4 VPN Concentrators experienced this issue. Reboot of those hubs resolved the issue for now.

[tele28312]

Hello, we experience the same issue even after reload, i wonder if there is any new update or ETR regarding this issue?

[SamerAl]

@tele28312

Please keep an eye on status.meraki.com

[SamerAl]

Whoever is experiencing this issue, you can restore service by rebooting the appliance in Passthrough Mode via the Meraki dashboard.

For steps to reboot your device please follow the below knowledge base article:
https://documentation.meraki.com/MX/Monitoring_and_Reporting/Using_the_MX_Live_tools#Reboot_Applianc...

[Felix_moreno]

Is rebooting the official "fix"? or is there supposed to be fix pushed out? 

[SamerAl]

@Felix_moreno , We have identified a proximate cause for the Meraki Auto VPN issues and are working on a remediation plan to restore normal service.

The reboot is a temporary fix, pelase try not to make any changes after the reboot ( for example adding new spokes/hubs) as this might break things again.

Please keep an eye on status.meraki.com for furthr updates regarding the permenant fix.

if your issue is urgent and massively impacting the business please try to get in touch with our support team.

You can find our 24/7 support number when you go to the dashboard? > Get help and cases  > Pick a Tile to Contact Support > Call Support team

[JessIT1]

Our MX95 looks to be allowing VPN connections now, we were down for awhile this morning with users getting L2TP connection attempt failed because the security layer encountered a processing error during initial negotiations with the remote computer.

[GIdenJoe]

A customer of mine also had a stop of traffic from Azure vMX to the branches.  The VPN's then showed a grey circle.

After a simple reboot appliance the tunnel resumed operation after 2 minutes.

[Knowguy]

What is super concerning here is that Meraki took down our MXs to multiple customers causing outages to 30+ sites without as much as asking.

This must be a very serious issue to be doing something this disruptive. Never seen anything this bad happen in awhile.

I also find that its not Passthrough or Concentrator MXs that were affected. Appears to be random. And its not "Auto VPN" its "ALL" IPSec which was finally admitted. 

Our environment was up and working until we were forcibly taken down when the fix was identified somewhere between 11 - Noon EST. Bad enough that it couldn't wait until off hours outside of 8:00 - 5:00.

So I wish they would change the wording to be something other than "AutoVPN" outage. We have a lot of Non-Meraki IPSec tunnels that were taken down today.

[Crocker]

Throwing in a +1 here. We were fine until around 10:00 AM Central Time, then everything broke/blipped.

Luckily enough, we had one of four VPN concentrators running 18.107.X due to an issue with SSID tunneling on MX 18.2, so it picked up the load until we could reboot its buddies which are all on MX18.2.11.2

[SamerAl]

Hi @Knowguy,

Sorry to hear that your customers have been affected by this incident,this should be resolved now and if any of your customers requested an RCA please try to log a case with our support team then we will try to provide you with that.

[Kevin_R]

Hello everyone,

Engineering has begun the process of rolling out the fixes to MXs that we have identified as still impacted by the problem (these would be passthrough MXs that have not yet been rebooted). We appreciate your patience and the rollout should be completed soon.

Status updates can be found at status.meraki.net

[haupt · Accepted solution]

This incident is now resolved. If you are still experiencing any issues, please contact Meraki Customer Support.

[Knowguy]

I hope we can get an explantion for today's events to get over to our customers.

[Crocker]

When can we expect the postmortem on this? What steps will be taken to try to ensure this doesn't happen again?

[SamerAl]

@Crocker Our team is conducting a thorough investigation and a robust review to ensure an accurate and comprehensive analysis. This process may take some time to complete.

If you have a support case please try to request the RCA on the case, if you do not have a case already logged, pelase try to log one by going to to the dashboard? > Get help and cases.