Issues with AutoVPN (incident)

Solved
SamerAl
Meraki Employee
Meraki Employee

Issues with AutoVPN (incident)

Hello community,

We'd like to inform you of an ongoing incident affecting AutVPN it is affecting passthrough MXs on 18.2.x, We are keeping status.meraki.com updated as we continue our remediation work.

If you are experiencing this issue, please try to reboot the MX as this might help resolve the issue.

 

Best Regards 

 

Samer 

If you found this post helpful, please give it kudos. If my answer solved your problem, click "accept as solution " so that others can benefit from it.
1 Accepted Solution
haupt
Meraki Employee
Meraki Employee

This incident is now resolved. If you are still experiencing any issues, please contact Meraki Customer Support.

View solution in original post

26 Replies 26
DarrenOC
Kind of a big deal
Kind of a big deal

Hi @SamerAl , for clarity, is that reboot all MXs involved in the auto-VPN or would a reboot of the main Concentrator resolve the issue?

Darren OConnor | doconnor@resalire.co.uk
https://www.linkedin.com/in/darrenoconnor/

I'm not an employee of Cisco/Meraki. My posts are based on Meraki best practice and what has worked for me in the field.
Shubh3738
Building a reputation

After Reboot the VMx Services started.

 

But after the sometime issue is happened again.

SamerAl
Meraki Employee
Meraki Employee

@Shubh3738 did you make any changes after the reboot( added a hub/spoke ) ?

 

if this is the case it might break again, our Team is still working on a permanent fix for the issue, so please keep an eye on status.meraki.com , if your issue is urgent and massively impacting the business please try to get in touch with our support team.


You can find our 24/7 support number when you go to the dashboard? > Get help and cases  > Pick a Tile to Contact Support > Call Support team

 

 

If you found this post helpful, please give it kudos. If my answer solved your problem, click "accept as solution " so that others can benefit from it.
SamerAl
Meraki Employee
Meraki Employee

@DarrenOC , I would recommend rebooting the Concentrator first as the issue is affecting MX's in passthrough.

If you found this post helpful, please give it kudos. If my answer solved your problem, click "accept as solution " so that others can benefit from it.
RaymondH
Here to help

Rebooting only the Hub MX resolved the issue for us, at least for now. Hope it lasts..

martin-netx
Getting noticed

We found that a restart of the vMX only was enough to bring back the VPNs.

Shubh3738
Building a reputation

Iam not making any changes.

 

Just waiting for the confirmation from Meraki side to resolve that issue permanently.

SamerAl
Meraki Employee
Meraki Employee

The team Identified the issue and they are working hard to roll out a permenant fix, I will update this thread when the issue is resolved.

 

 

If you found this post helpful, please give it kudos. If my answer solved your problem, click "accept as solution " so that others can benefit from it.
finzwake
Conversationalist

Tunnels dropped specifically to our VPN concentrator Hub (running 18.211.2) around 1:06am EST. MXs in routed mode don’t appear affected.  Tunnels appeared up on VPN Status page for all MXs.

 

Rebooting the Hub concentrator restored tunnels, for now

RaymondH
Here to help

About same experience, tunnels seemed to be up, tracert worked, but no dns, ping or Umbrella connection on the spoke MX's.

Felix_moreno
Conversationalist

2 out of my 4 VPN Concentrators experienced this issue. Reboot of those hubs resolved the issue for now.

tele28312
Just browsing

Hello, we experience the same issue even after reload, i wonder if there is any new update or ETR regarding this issue?

SamerAl
Meraki Employee
Meraki Employee

@tele28312

 

Please keep an eye on status.meraki.com

If you found this post helpful, please give it kudos. If my answer solved your problem, click "accept as solution " so that others can benefit from it.
SamerAl
Meraki Employee
Meraki Employee

Whoever is experiencing this issue, you can restore service by rebooting the appliance in Passthrough Mode via the Meraki dashboard.


For steps to reboot your device please follow the below knowledge base article:
https://documentation.meraki.com/MX/Monitoring_and_Reporting/Using_the_MX_Live_tools#Reboot_Applianc...

If you found this post helpful, please give it kudos. If my answer solved your problem, click "accept as solution " so that others can benefit from it.
Felix_moreno
Conversationalist

Is rebooting the official "fix"? or is there supposed to be fix pushed out? 

SamerAl
Meraki Employee
Meraki Employee

@Felix_moreno , We have identified a proximate cause for the Meraki Auto VPN issues and are working on a remediation plan to restore normal service.

 

 

The reboot is a temporary fix, pelase try not to make any changes after the reboot ( for example adding new spokes/hubs) as this might break things again.

 

Please keep an eye on status.meraki.com for furthr updates regarding the permenant fix.

 

if your issue is urgent and massively impacting the business please try to get in touch with our support team.


You can find our 24/7 support number when you go to the dashboard? > Get help and cases  > Pick a Tile to Contact Support > Call Support team

If you found this post helpful, please give it kudos. If my answer solved your problem, click "accept as solution " so that others can benefit from it.
JessIT1
Building a reputation

Our MX95 looks to be allowing VPN connections now, we were down for awhile this morning with users getting L2TP connection attempt failed because the security layer encountered a processing error during initial negotiations with the remote computer.

GIdenJoe
Kind of a big deal
Kind of a big deal

A customer of mine also had a stop of traffic from Azure vMX to the branches.  The VPN's then showed a grey circle.

 

After a simple reboot appliance the tunnel resumed operation after 2 minutes.

Knowguy
Getting noticed

What is super concerning here is that Meraki took down our MXs to multiple customers causing outages to 30+ sites without as much as asking.

This must be a very serious issue to be doing something this disruptive. Never seen anything this bad happen in awhile.

I also find that its not Passthrough or Concentrator MXs that were affected. Appears to be random. And its not "Auto VPN" its "ALL" IPSec which was finally admitted. 

Our environment was up and working until we were forcibly taken down when the fix was identified somewhere between 11 - Noon EST. Bad enough that it couldn't wait until off hours outside of 8:00 - 5:00.

So I wish they would change the wording to be something other than "AutoVPN" outage. We have a lot of Non-Meraki IPSec tunnels that were taken down today.

Crocker
A model citizen

Throwing in a +1 here. We were fine until around 10:00 AM Central Time, then everything broke/blipped.

 

Luckily enough, we had one of four VPN concentrators running 18.107.X due to an issue with SSID tunneling on MX 18.2, so it picked up the load until we could reboot its buddies which are all on MX18.2.11.2

SamerAl
Meraki Employee
Meraki Employee

Hi @Knowguy,

 

Sorry to hear that your customers have been affected by this incident,this should be resolved now and if any of your customers requested an RCA please try to log a case with our support team then we will try to provide you with that.

If you found this post helpful, please give it kudos. If my answer solved your problem, click "accept as solution " so that others can benefit from it.
Kevin_R
Meraki Employee
Meraki Employee

Hello everyone,

 

Engineering has begun the process of rolling out the fixes to MXs that we have identified as still impacted by the problem (these would be passthrough MXs that have not yet been rebooted). We appreciate your patience and the rollout should be completed soon.

 

Status updates can be found at status.meraki.net

If you found this post helpful, please give it kudos. If my answer solved your problem, click "accept as solution" so that others can benefit from it.
haupt
Meraki Employee
Meraki Employee

This incident is now resolved. If you are still experiencing any issues, please contact Meraki Customer Support.

Knowguy
Getting noticed

I hope we can get an explantion for today's events to get over to our customers.

Crocker
A model citizen

When can we expect the postmortem on this? What steps will be taken to try to ensure this doesn't happen again?

SamerAl
Meraki Employee
Meraki Employee

@Crocker Our team is conducting a thorough investigation and a robust review to ensure an accurate and comprehensive analysis. This process may take some time to complete.

 

If you have a support case please try to request the RCA on the case, if you do not have a case already logged, pelase try to log one by going to to the dashboard? > Get help and cases.

If you found this post helpful, please give it kudos. If my answer solved your problem, click "accept as solution " so that others can benefit from it.
Get notified when there are additional replies to this discussion.
Welcome to the Meraki Community!
To start contributing, simply sign in with your Cisco account. If you don't yet have a Cisco account, you can sign up.
Labels