Client VPN Error After January Windows Updates

[Darcon]

After patching a couple machines with the January Windows Updates I found that our Meraki Client VPN no longer works. I get error 789 in Windows Event Viewer, and nothing at all in the Meraki Event Log.

This is the error displayed when attempting to connect. It errors out within a second of hitting connect, doesn't even prompt for creds.

We've had AssumeUDPEncapsulationContextOnSendRule set to 2 for some years. I tried removing / rebooting / re-adding this key; same error. I also tried switching our PSK to a key without special characters, same error.

I have confirmed this on 2 different machines. Unpatched machines continue to connect without issue.

Anyone else run into this today? 

[BrandonS]

Not today (yet), but pretty much every Windows update brings me service tickets about broken client VPN and 100% of the time they are resolved by deleting and creating the VPN again.  There is probably a quicker way and an underlying reason, but I have not cared to dig further personally..  

EDIT: Apparently this time requires removing a Windows update.  Keep scrolling..

[Darcon]

I gave re-creating the connections a try, no luck. I also tried running through my mobile hotspot to no avail.  

*OS is Windows 10 20H2

**After removing the updates (and doing nothing else) the VPN works again. 

[RobertMiranda]

Which KB updates did you uninstall? Having same issue now. Thank you.

[Darcon]

I uninstalled KB5009543 and KB5008876. I'll see if I can narrow it down tomorrow. 

[RobertMiranda]

Thanks. I believe it's KB5009543 but waiting for a response from my users to confirm. Once I do, I will post here as well.

Running this to uninstall:

wusa /uninstall /kb:5009543

[Darcon]

I've seen KB5009543 as well. Other platforms that use L2TP are being impacted too. 

[Ben0391]

worked for me as well. 
The Win11 Update should be KB5009566

[Jawson]

Woke up here in UK with same slowly hitting all our users. Seems to be anything using PAP auth. Other L2TP VPNs we have using MSCHAP seem fine. Removing update fixed for now.

[RobertMiranda]

This is working for me. Confirmed on a few laptops this morning.

[RobNYC7]

---

@RobertMiranda wrote:

wusa /uninstall /kb:5009543

---

This is a life saver, thanks! We had over 15 people and counting affected this morning.

Make sure you run your prompt as admin.

[BeckerIT]

It is KB5009543. I just spent 4 hours cleaning up the mess from the installation of that update.

[BreckpointIT]

You are a lifesaver! Just got the Client VPN set up and it was working, then boom... nothing. Thanks!

[April_Fulton]

The uninstall did not work for me. I received an error stating KB5009543 is required by your computer and cannot be uninstalled.  Help!

[BigMtnIT]

Having the same issue. KB5009543 2022-01 Cumulative Update breaks L2TP on Win10 computers.  I have about 15 VPNs and they all give the same error now as Darcon's image above.  Recreating does NOT fix this.  Uninstalling KB5009543 does "fix" it but at some point we will all want this update and probably get it whether we want it or not.  Maybe Meraki can start lobbying Microsoft now since this will begin to propagate very soon and show up on more and more customers.  If anyone finds a workaround please post.  Thanks.

[RobertMiranda]

Thanks for confirming. Have 3 users now, but tomorrow morning it will go 10X. Fun times.

[BlakeRichardson]

I dealt with a similar issue last year, I have no idea why MS seem to like breaking VPN connections. 

[KumarKaliappan]

Same issue for my clients. VPN connection is not working after installing the 2022-01 CU.

[Nikola]

We've been also affected. Both win10 and win11. 

Recreating connections doesn't help.

Is Cisco doing anything to resolve the issue? We cannot go forever pausing and removing updates.

[GiacomoS]

Good morning awesome people of the Community,

We have observed this as well throughout the day. As you have rightfully identified, this seems to be related to KB5009543. As this is a Microsoft update that is breaking the Windows VPN adapter, we are unlikely going to be able to affect it at this stage, but we are investigating internally nonetheless. 

The recommendation at the moment is to uninstall that update and let Microsoft know.

I've seen mention of other vendors being affected as well; @Darcon could you confirm if you have experienced this directly?

Many thanks!

Giac

[Tiainen]

Hi all,

We have the same issue. I can confirm uninstalling KB5009543 fixes the issue.

I've just opened a support ticket to Microsoft, #‎29198981‎.

[vhovan]

Hello,

I have a virtual windows 10 that i use for VPN connection to different clients, update broke the VPN and i uninstalled the update and its all good, however i noticed that VPN clent connections to MX80 is working fine only MX64 is broken!! could it be Meraki OS issue??

[Avenir]

We have issue with MX100 and virtual MX100

[GiacomoS]

@vhovan it sounds like you may be encountering a different problem here. I would recommend following the Troubleshooting Client VPN KB in the first place, and if you are still having issue afterwards, follow up with a Dashboard case with details of the error, troubleshooting done, packet captures if possible and timestamps of the attempts, so we can help further.

Many thanks!

Giac

[Darcon]

@GiacomoS I am internal IT and we only use Meraki gear, so no first-hand knowledge. I've just seen other affected platforms mentioned online. 

[GiacomoS]

Thank you @Darcon , I appreciate your input!

[iReidy]

It worked for me, uninstalled

Windows 10
(KB5009543)
(KB5008876)

Windows 11
KB5009566

[MatiRufa]

I have Windows 11 and I get this error 0x800f0905

[JimmyM]

Same Problem with multiple customer. Uninstall KB5009543.

[kkwok]

Did it fixed the issue, i just tried with a user no dice.

[AntonioGuerrisi]

Users are not administrator on Device, and they are outside the network for verify with DC our administrator cred.

There is a way to terminate the Meraki VPN without the windows built-in client? OpenVPN works on it? 

[Netmgr-24]

Exact same issue here on Windows 11.  Updated machine first thing this am, and have been trouble shooting since.  Led me here.

Event log:
The user SYSTEM dialed a connection named [XXXX] which has failed. The error code returned on failure is 789

Have tried the usuals - resetting all network connections, delete and recreate vpn client, etc...

January 11, 2022—KB5009566 (OS Build 22000.434) (microsoft.com)

Removed update with command:

DISM /remove-package /online /PackageName:Package_for_RollupFix~31bf3856ad364e35~amd64~~22000.434.1.4

[Netmgr-24]

After removing on Win 11, connected to Meraki (could see attempt in logs), but still failing with 720 error.  To fix this, had to uninstall WAN Miniport (L2TP) under Network Adapters in Device Manager.  Then scan for changes and it will reinstall.  Then connected fine.

Having some issues with routing to internet once connected on the Meraki side, but fixed by updating routes so non-vpn traffic uses local ISP, not VPN.

Working again!

[dylan-mn]

We are having the same issue, I can confirm wusa /uninstall /kb:5009543 fixes it for now at least.

[KennyH]

We had the same issue, thanks for the direction.  Also, anyone know of a status page for Meraki services or a updates page to subscribe to?

[GiacomoS]

Hey @KennyH , 

The Service Notices board is a good place to start!

Hope this helps!

Giac

[jcgvt]

Any eta...this will turn in a big deal for compliance reasons quickly.  The answer CANNOT simply be rollback the MSFT patch or don't install it.

[Darcon]

I think this is more of a MS problem than a Meraki problem. Given how widespread this issue is I would expect new patches to be released within a week or so. 

[jcgvt]

Agreed.  Bigger than Meraki. 

[Marsrock2021]

It's only a Microsoft problem because cisco won't roll out a vpn client for meraki. The free windows vpn is really only for devices you get free from isps where its not worth their while to do anything better. To have systems with 1000s of users not having a vendor client is a joke tbh.

[AntonioGuerrisi]

MX100 and upper isn't for manage tens people. The joke is that for 10k+ $ firewall we don't have Meraki client or a workaround to use third-party client. Use only a single built-in client is a point of failure that Cisco have to resolve if want to sell to bigger players. 

[OVERKILL]

AnyConnect IS available on the Meraki platform, currently without a fee (because you need to be running 16.xx) but there will be a charge once the 16-series because stable. 

[Pharkurnell]

Specially with so many working from home due to Covid...

[LM-TECH]

wusa /uninstall /kb:5009543

worked for me on Win10

[Pharkurnell]

Love from Microsoft...

[Parsoli]

What's interesting to us is, we have two different companies using Meraki MX's, both experiencing the same issue as everyone else above, but none of the impacted workstations/laptops have those patches installed.  Validated using the wusa /uninstall /kb:5009543 and the patches are not found, nor seen in Update history or via the Ivanti patch deployment tools we use.

[Darcon]

The KB's may be different depending on which OS you are running. Try going into Windows Settings on an affected machine, then Windows Update, click on "View Update History". Write down whatever KB's were installed yesterday or today. 

[mikeTNYC]

Same problem on a few systems today. Uninstalling KB5009543 fixed it. 

[delfuego]

This is a mess. MS keeps screwing up. Starting to feel like the old days with Windows Updates.

Thanks for the help. Removing  KB5009543 working on a few machines already.

[Marsrock2021]

Find it strange that people blame Microsoft. The built in vpn is no solution for business level vpn. The problem is that cisco bought meraki in 2012 and 10 years later they have no vpn client of their own. The whole cisco anyconnect debate going on for as long. The blame for this lies solely with the guys selling the hardware. Windows client is for $200 devices not 10 grand devices. The anyconnect is in beta for god knows how long on meraki and still no dates for GA. 

[Mike6116]

+1 on this,    in my case 100K on Meraki HW  and have to deal with this kind of BS, really sucks...

[ShadowoftheDark]

gawddammit, i knew this was related to the windows update. one of our machines borked when i uninstalled the updates. fkn hell all of our vpn users are using windows 10 laptops

[Mike6116]

Greetings from Mexico,  same issue here, some PC´s have the KB5009543  installed  , all having the same error,  we pulled back that update and everything works fine now....

[JoshGesite]

Uninstalled KB5009543 on one machine and worked properly. Have another machine where I uninstalled KB5009543, which removed the Error 789, but now getting Error809. Anyone else running into this issue?

[delfuego]

Try a reboot for the 809 error. I tested to see if the rollback worked without restart (even after prompted) on a machine and got the 809 event.

[JoshGesite]

Tried a reboot. Also did the steps on this page: https://documentation.meraki.com/MX/Client_VPN/Guided_Client_VPN_Troubleshooting -- such as the regedit. Also disabled the Xbox Live Network Service and other steps here: https://www.thewindowsclub.com/troubleshoot-vpn-error-809-on-windows 

but still have not resolved the issue.

[Pharkurnell]

like millions of others have been struck with this one on many laptops. Great issue to have with 80 laptops around Australia...

My issue is uninstalling the update.

Logged in as local admin  I right click on KB5009543 and there is no uninstall option ???

I also tried wusa /uninstall /kb:5009543 but it tells me its a required update you cannot uninstall it...

Thoughts?

[ShadowoftheDark]

must be a policy m8, wusa /uninstall worked for me. i'm on my 5th today lol

[Lourdes]

We are also having an issue with the VPN connection but some users did not update their windows but were suddenly unable to connect to VPN. We are using windows 10 and there is no KB5009543 at KB5008876. 

What to do?

[ShadowoftheDark]

is the authentication set to username / password?

in the security tab of the vpn adater is it set to required - only thing checked is PAP?

[Lourdes]

Yes, the authentication is set to username/password.

VPN type is L2TP/IPsec with a pre-shared key. 

The PAP and CHAP ver 2 is actually checked.

[ShadowoftheDark]

i think its only PAP, chap is unchecked  

Client VPN OS Configuration - Cisco Meraki

[Itchriskentas]

for some reason it both connects and if you try to change it and put it as meraki says and check only pap it deletes instanly the user name and pass so you cant  do anything ... tyried to unistall the updates nothing still the same mc is the best ... 

[MateTamas]

I had the same problem, uninstalling that update + restart solved it.

Where can I raise a ticket to MS? The more opened ticket the sooner solution... 🙂

Thank you!

[Tiainen]

Hi,

I guess if you don't have access/account to admin.microsoft.com site, then this is the way:

https://support.microsoft.com/en-us/windows/send-feedback-to-microsoft-with-the-feedback-hub-app-f59...

[delfuego]

Update downloaded and installed itself again on one machine today (after removal yesterday).

[crabwebs]

follow this article to block (hide) the update:  https://www.maketecheasier.com/hide-updates-in-windows-10/

[Pharkurnell]

I believe this is the case here too... but running that powershell solution below (Id never seen before but bookmarked it!) to block update on 60-80 remote computers isn't an option for me.

[Nikola]

I think this is a serious issue for many users and Cisco should handle this with Microsoft, not ask users to submit tickets. 

[JimmyM]

Does anyone know if Meraki work with MS with that or if someone has opened a case with Microsoft.

I can open an MS ticket for that but I don't want to do a thing than someone is already doing.

Regards,

[delfuego]

Thanks again for the help. If anyone has a permanent solution or what exactly the update changes, please let us know. This has been happening all over for us.

[JimmyM]

I opened a support case with Microsoft, i will give you a follow up.

[360IT]

Thanks to everyone for your combined efforts in troubleshooting this latest vpn client issue, what a fiasco.

I can confirm that removing KB5009566 on Win11 resolved the issue for me. I lost 13vpn configs on my machine yesterday, a pain to say the lease. Now off to do a regression patch on a mind numbing number of client systems, Gack! 

I have been dealing with random windows client vpn connection issues on MX devices for over a year now. Have tried all the methods to resolve, not one consistently is the fix however. One method I have tried which is partially documented is the removal of all wan miniports, but before scanning for hardware changes to re-install I would configure the required vpn's and when finished I would then re-install the miniports. 

Have no idea why, but that process has worked on quite few win 10 machines over the past several months.

[MarkChan]

Microsoft acknowledged the issue. I am not sure if the workaround exist for Meraki MX.

The description is as below:

After installing KB5009543, IP Security (IPSEC) connections which contain a Vendor ID might fail. VPN connections using Layer 2 Tunneling Protocol (L2TP) or IP security Internet Key Exchange (IPSEC IKE) might also be affected.

Workaround: To mitigate the issue for some VPNs, you can disable Vendor ID within the server-side settings. Note: Not all VPN servers have the option to disable Vendor ID from being used.

Next steps: We are presently investigating and will provide an update in an upcoming release.

Affected platforms:

• Client: Windows 11, version 21H2; Windows 10, version 21H2; Windows 10, version 21H1; Windows 10, version 20H2; Windows 10, version 1909; Windows 10, version 1809; Windows 10 Enterprise LTSC 2019; Windows 10 Enterprise LTSC 2016; Windows 10, version 1607; Windows 10 Enterprise 2015 LTSB
• Server: Windows Server 2022; Windows Server, version 20H2; Windows Server 2019; Windows Server 2016

Windows 10, version 21H2 | Microsoft Docs

[MarkChan]

Microsoft released an Out-of-band patch:

Windows 10 - KB5010793

Details Here

Option 1:

Run Windows Update, KB5010793 will appear under optional download.

Option 2:

Download the patch from there: Microsoft Update Catalog

Please download the matching Windows 10 Version.

Windows 11 - KB5010795

Details Here

Option 1:

Run Windows Update, KB5010795 will appear under optional download.

Option 2:

Download the patch from there: Microsoft Update Catalog

[CharlieCrackle]

Microsoft states that it may be possible to mitigate the bug by disabling the 'Vendor ID,' if possible, on the VPN server.   Meraki is this possible by settings or a firmware update   as this update is just going to reinstall.

Microsoft confirmed on Thursday that "Certain IPSEC connections might fail" and that they will fix the issue in an upcoming release of Windows.

[Pharkurnell]

...and that they will fix the issue in an upcoming release of Windows.

Thats all well and good, but what about the 1000's of people who are being forced to work from home due to Covid restrictions.

They need to release it ASAP!

[techmoc]

I've been testing a W10Pro Client VPN with the update against different MX firmware versions 14.xx/15.xx/16.xx and found that Client VPN is working against MX 14.56 on a Z1 even though it's not against newer releases on MX appliances, so @GiacomoS it might be worthwhile to review what changed on the Client VPN code from firmware MX 14.56 vs the newer ones that are not working.

Can someone else test Client VPN with update working against MX 14.xx to confirm?

I would expect a faster fix from MX firmware Dev/Support team vs waiting for Microsoft to release a hotfix that would have to be deployed or waiting it to be rolled in a future update.

[GiacomoS]

Hey superheroes of the Community,

Thank you for all the info you are sharing and all your testing. 

We spotted the update from Microsoft as well and have been discussing it internally, but at the moment we have no major feedback to add aside from expecting Microsoft to fix what they changed with the Vendor-ID. 

@techmoc to answer your question, there has been a major overhaul of client VPN between the 14 and 15 releases and the Vendor-ID field is not sent on the 14 release. 

I do not believe this behaviour is due to change on current and future firmware trains, but we are continuing conversations internally around the matter.

Hope this helps!

Giac

[stefbauer]

Any chance there is a fix to 15.x to allow control of the vendor id being passed (or not)?

[CharlesIsWorkin]

Thanks! Looking forward to what you guys come up with!

For those interested, here is the link to the issue addressed in the update by Microsoft.

https://docs.microsoft.com/en-us/windows/release-health/status-windows-10-21h2#2773msgdesc 

[Imperio]

Seems to be our temporary fix 👀

[LehmansITMgr]

Has anyone pursued mitigations such as...

• Installing the 16.14 release candidate (on a production machine) and using the Cisco AnyConnect VPN client?
• Standing up another MX with the older firmware (that isn't susceptible to this problem) just for client VPN connections?

Meraki: What are your thoughts on the above?

[GiacomoS]

Hey @LehmansITMgr ,

My current understanding is that Anyconnect is unaffected, so that should be a valid workaround.

Option two I'm be a bit more reluctant to recommend, as release 14 is no longer publicly available (except for legacy hardware), and would also require an additional MX, with all the headaches it entails (rearchitecting, licensing, etc.)

Hope this helps!

Giac

[eyre-jr]

@GiacomoS 

Isn't AnyConnect on Meraki now subject to AnyConnect licensing though (i.e. you must have valid AnyConnect licenses to use)? I was looking yesterday and where I previously recall there was a period where it was permitted to use without license(s), the documentation now all seems to point to licenses required (though potentially not enforced)

[LehmansITMgr]

Hi @GiacomoS 

RE: using AnyConnect, one very important question remains: 16.14 is a release candidate (not GA yet).  Is it irresponsible to install an RC on a production machine?

[GiacomoS]

Hey team,

@eyre-jr , yup! To quote the KB :
Customers are required to have a valid AnyConnect license to use AnyConnect with the MX Appliance. Customers are not required to validate their licenses via the Meraki MX or the dashboard. Customers will only be required to accept terms and conditions of use before they can enable AnyConnect.

@LehmansITMgr , that is a "Depends" question 😁 . Each of your businesses is different and what may work for you, may not necessarily work for someone else. 
I shared some thoughts around the different types of firmware in the Community in the past, here. Whether or not implementing GA software is the right choice for your company is really up to you. 

I will always be an advocate for having a maintenance window that allows you to upgrade and test a bit of everything, so that you always have room to revert if anything goes wrong. 

Also, make sure you take a gander at the release notes, so you are aware of what you can expect. 

Giac

[LehmansITMgr]

@GiacomoS 

Having a maintenance window to test and roll-back if necessary makes sense.  However, I'm very concerned about the stability of the 16.14 version, since the last bullet point in the "known issues" in the release notes states: "There is an increased risk of encountering device stability issues on all platforms and across all configurations."

I have an open support ticket about this bullet point to see if a technician can explain what the risk is in more detail.  (Case 07499526)

[CharlesIsWorkin]

I just had it happen again to the same user. 😞

When you install an update, does it get reinstalled the next day? This is a home laptop that a user is using to connect.

[Imperio]

You could disable a particular update.

Find the necessary tool here:
https://m.majorgeeks.com/files/details/wushowhide.html

[CharlesIsWorkin]

Thanks!

[eyre-jr]

For those looking for a direct MS download, you can find it here:

http://download.microsoft.com/download/F/2/2/F22D5FDB-59CD-4275-8C95-1BE17BF70B21/wushowhide.diagcab

EDIT: some browsers (Chrome for me) may not start the download, if that's the case, try pasting into the address bar, rather than clicking the link.

In testing, I've seen a single case where it didn't show the list of updates to hide immediately after restarting post-update removal, in that instance, restarting the computer a second time then had the tool show the list of updates as usual.

i.e. process steps are:

1. wusa /uninstall /kb:5009543 (from an elevated command prompt, KB 5009543 is for Windows 10, replace with 5009566 for Windows 11)
2. Restart when prompted
3. Run wushowhide.diagcab and hide the relevant update (likely at the bottom of the list), if no updates can be found, attempt restarting again, and re-run wushowhide.diagcab after restart.

If the update hasn't yet installed, you can skip straight to step 3.

[RyanMcNetworks]

Exact same error here.  The user said they just did windows update and we use Meraki for client VPN.  One of our techs is walking them through rollback now.

[nnhood]

Hello,

This is a thread I found talking about replacing a .dll which allows the update to take place and still work with the VPN.

Windows 11 Update (KB5009566) inhibits VPN connection - Page 4 - Microsoft Tech Community

It's about halfway down the page, I think I'd rather wait for an update that fixes this or a nice Meraki VPN client (HINT HINT)

Figured I would throw this out there though.

Thanks,

M.

[eyre-jr]

That's an interesting find - whilst as you point out it's still a long way from ideal, the implication would be that the update could be applied, therefore patching a number of vulnerabilities, whilst still maintaining VPN functionality (albeit, I suspect, at the expense of at least some of the IKE vulnerabilities remaining unpatched, but given not applying the update at all would mean that the IKE vulnerabilities, PLUS the other vulnerabilities would be unpatched, it certainly seems the lesser evil)

[nnhood]

We only have 30 some employees at our company, I'm glad I'm not working at a school district anymore where there was 1500, I guess not everyone needs VPN though just depending on the company.

Thanks for the Kudos!  I would think they would correct this quickly or hope they would at least.

Matt

[360IT]

I have tested this on a system with KB5009566 installed, I used a version of the IKEEXT.DLL file dated 11/15/2021.

Booted the system with Hiren's and overwrote the dll dated 1/12/2022. Rebooted, tested VPN and works no issue. Not a practical solution for remote deployment, but if there is local access it works allowing the KB5009566 update to be installed. Not sure if subsequent updates will replace the ikeext.dll file, will continue testing on a bench system and post new information should it be helpful. Fingers crossed for a real solution from Meraki and/or Microsoft. 

[louyo]

+1, worked here on both W10 and W11

[rschandl]

Thanks nnhood this was very helpful. I created a PowerShell script which I was able to deploy as a Win32 app via Intune. Not sure if this was the best approach, but I like it better than rolling back the cumulative updates. Here's my script for anyone interested:

Stop-Service -Name "IKEEXT" -Force

(Get-Service -Name "IKEEXT").WaitForStatus('Stopped')

$acl = Get-Acl C:\Windows\SysNative\IKEEXT.DLL
$AccessRule = New-Object System.Security.AccessControl.FileSystemAccessRule("BUILTIN\Administrators","FullControl","Allow")
$acl.SetAccessRule($AccessRule)
$acl | Set-Acl C:\Windows\SysNative\IKEEXT.DLL

Copy-Item "$PSScriptRoot\IKEEXT.DLL" -Destination "C:\Windows\SysNative"

Start-Service -Name "IKEEXT"

[MikeL2022]

Just confirming that the removal of the Wed 12th MS update KB5009543 for WIN10 worked for me for MX64.

as suggested by JP Harden Sr., Cisco Meraki Support 

https://community.meraki.com/t5/Meraki-Service-Notices/Microsoft-Windows-update-breaking-Client-VPN/...

[JohnMeloche]

Also having the same issue.

[ChristopherLDR]

What is the best way to track when this issue is resolved (i.e. when Microsoft fixes their update)?  I would like to avoid having to update and remove the updating randomly until the item is resolved.

[AntonioGuerrisi]

You can check into Microsoft Release Health page

https://docs.microsoft.com/en-us/windows/release-health/status-windows-10-21h2#2773msgdesc

I think you can see here every update

[ShadowoftheDark]

Anyone else here getting the L2TP even with KB5009543 uninstalled? 

I think the KB5008212 is also messing up the client vpn. 

I uninstalled it and my client was able to connect.

[Acider]

We are on Windows 10 and L2TP works for us after removing KB5009543. Just be sure to restart the computer, then use wushowhide.diagcab to hide this update.

[ShadowoftheDark]

just had to uninstall from 24 laptops. mondays are going to be hell until this update is fixed.

[Duke_Nukem]

This hit my pilot WUFB group (15 laptops).  Thankfully didn't release it to the masses (~400 laptops).  What's interesting is I'm not seeing the error connecting the windows VPN to my DR site.  That site is running an older MX80 (14.56), but it has the PCI compliant VPN settings in place on the MX.  So my VPN client connects without issue having the latest MS patch installed. (AES256, SHA1, DH14, IKEV1).  Our main VPN site is running an MX100 (15.44) but it does not have the PCI compliant settings in place (couldn't get it to work correctly with our CMAK VPN client a few months back so we had the tech back it out).  

Just some more data to add to the pile...

[Scottdb]

I just realized that the only people that were affected by the update kb5009543 are the users that connect to the MX84 VPN. none of the users which I could count on two hands have had the issues connecting to MX64 or MX80's. 

Wonder what's with that.

[MikeL2022]

I can confirm that my MX64 clients had the issue, only the ones that had the update installed. some did not have the update installed due to their individual PC settings.

[Scottdb]

Interesting my MX64 works with the patch installed the only other thing that is different is I don't have a 2FA service set on that one at this time.

[eyre-jr]

Firmware versions the same across your different Models?

GiacomoS has confirmed 14.x firmware versions don't use the Vendor ID (Which Microsoft have said disabling is a workaround to the issue) whereas from 15.x Vendor ID is enabled (and thus don't play well with the MS updates)

[LehmansITMgr]

I can confirm that the MX100 is also affected.

[janguiano]

I can confirm that uninstalling kb5009543 resolves the issue. We have mx67 and mx67w ones.

[rsafadi911]

the problem here is that windows will not issue a fix for this problem until feb 8th patch tuesday

so cisco needs to take ownership of the problem and disable Vendor ID from there server side vpn config.

and then push that update to all mx devices. With everyone working from home as an MSP we have over 200 users effected.

[AntonioGuerrisi]

Microsoft has never sayd the will release a fix in the normal path.

Internal contact say us that Microsoft will release a fix ASAP but they still have problems with the deploy.

We don't know if is dev issue or a technical problem with Windows update, but when the update is ready will be released immediatly

[MikeL2022]

something to consider for supporting home workers: user's PCs may have different windows update settings setup such as auto download and install, download only or none at all. After uninstalling the update, the next update cycle may install it again depending on the PC settings.

[criceTrico]

Had the same issue today.  uninstalling KB5009566 for win 11 and KB5009543 on win 10 fixed it for me

[accmestep]

Same issue here.  Reinstalling the VPN did not resolve.  We are over 20 users that we have had to uninstall KB5009543 and KB500887 to allow them to proceed with the connection.

[stefbauer]

Uninstall of the KB worked for me - and was swiftly reinstalled by updates - attempting to pause that through intune now.

Anyone have any word from Microsoft other than they are aware - they are non-responsive.

Also - any input from the Meraki side on the ability to remove the vendor ID, seems to be the suggested "fix" from Microsoft.  They clearly are just pointing fingers - because why would they have an issue??!?! 

[CharlesIsWorkin]

This. You just wrote my feelings out. If the engineers of MS and Meraki could talk it out and figure out the reason for the Vendor ID issue, slap something together, and get things moving, I'd love to see that. I'd also like to hear about it with some clear communication from Meraki.

I hope everyone puts some Kudos on the OP and also submits a support ticket. Some collective customer pressure can go a long way.

[AlexLyon]

Yep, good question...! 

Also - any input from the Meraki side on the ability to remove the vendor ID,  

May be affraid by side effect ?  but if we ask on a case to remove for one organization the vendor ID ?  someone try ? 

[Duke_Nukem]

Having a hell of a time trying to post this.

This hit my pilot WUFB group (15 laptops).  Thankfully didn't release it to the masses (~400 laptops).  What's interesting is I'm not seeing the error connecting the windows VPN to my DR site.  That site is running an older MX80 (14.56), but it has the PCI compliant VPN settings in place on the MX.  So my VPN client connects without issue having the latest MS patch installed. (AES256, SHA1, DH14, IKEV1).  Our main VPN site is running an MX100 (15.44) but it does not have the PCI compliant settings in place (couldn't get it to work correctly with our CMAK VPN client a few months back so we had the tech back it out).  

Just some more data to add to the pile...

[OVERKILL]

Yes, the 14.xx releases aren't impacted because the vendor_ID flag isn't set in the firmware. That was added with the 15.xx series and that seems to be what is causing the issue on the Microsoft side. 

[eyre-jr]

See this reply  from GiacomoS above.

In short, Vendor ID isn't sent in 14.x but was added in 15.x

Disabling Vendor ID is Microsoft's published workaround, so it'd make sense any connections terminating on an MX running 14.x continue to work

[criceTrico]

How do you disable the vendor ID?  You talking on the radius server policy?

[eyre-jr]

It isn't (at least currently) an option on Meraki; it just happens to be that on older firmware versions it wasn't in use/sent

[Rules_2301]

I still have the problem, I already uninstalled the MS updates and I'm not left
wusa /uninstall /kb:KB500887
wusa /uninstall /kb:KB5009543

[Todd_S1]

This issue has been fixed:

https://support.microsoft.com/en-us/topic/january-17-2022-kb5010793-os-builds-19042-1469-19043-1469-...

[JimmyM]

tested and working!

[MarkChan · Accepted solution]

Microsoft released an Out-of-band patch:

Windows 10 - KB5010793

January 17, 2022—KB5010793 (OS Builds 19042.1469, 19043.1469, and 19044.1469) Out-of-band (microsoft...

Option 1:

Run Windows Update, KB5010793 will appear under optional download.

Option 2:

Download the patch from there: Microsoft Update Catalog

Please download the matching Windows 10 Version.

Windows 11 - KB5010795

January 17, 2022—KB5010795 (OS Build 22000.438) Out-of-band (microsoft.com)

Option 1:

Run Windows Update, KB5010795 will appear under optional download.

Option 2:

Download the patch from there: Microsoft Update Catalog

[RobertMiranda]

In case others are using WSUS and unable to add the KB5010793 to their WSUS server.

On the WSUS server with PowerShell run as Admin:

$WsusSrv = Get-WsusServer

$WsusSrv.ImportUpdateFromCatalogSite('b278111f-a855-412d-ba3a-26170fdb08eb', 'C:\Users\YourProfileName\Downloads\windows10.0-kb5010793-x64_3bae2e811e2712bd1678a1b8d448b71a8e8c6292.msu')

[Scottdb]

had the same issue before I checked here and it solved my problems just getting the ting to let edge download it. 

[julioleal]

Tested on windows 11 that had the problem and it solved the problem.

[Aztec_Ninja]

A bit little here but you could use the following script to help automate the uninstall - 

From an elevated cmd prompt or other script option: wusa /uninstall /kb:5009543

The long-term option,  upgrade MX to  ver16.x and install the long waited Cisco AnyConnect.  

[EvanKenzoD]

Hello, is there a solution for this for Windows 10 22H2 version? Seems the update on the link provide is not working on my side. Thanks

[MarkChan]

Microsoft released an Out-of-band patch:

Windows 10 - KB5010793

Details Here

Option 1:

Run Windows Update, KB5010793 will appear under optional download.

Option 2:

Download the patch from there: Microsoft Update Catalog

Please download the matching Windows 10 Version.

Windows 11 - KB5010795

Details Here

Option 1:

Run Windows Update, KB5010795 will appear under optional download.

Option 2:

Download the patch from there: Microsoft Update Catalog

[CharlesJN]

Thanks.

VPN works after being updated. Windows 10 21H2 here.

[Rules_2301]

I did not succeed

[Rules_2301]

sorry, WINDOWS 11

[DeepakRaiPune]

I was also getting the below connection error:

"The L2TP connection attempt failed because the security layer encountered a processing error during initial negotiations with the remote computer"

I checked Windows Event Viewer and found below log entry:

"CoId={452A7588-0950-0002-5660-36455009D801}: The user SYSTEM dialed a connection named N4mative which has failed. The error code returned on failure is 789."

I found below Windows Update was causing this issue. After uninstalling KB5009543 and restarting laptop my issue resolved.

[cmr]

Microsoft have released an out of band update to fix this.  KB5010795

You should be able to get it from Windows update and information is here: https://support.microsoft.com/en-us/topic/january-17-2022-kb5010795-os-build-22000-438-out-of-band-2...

[Itchriskentas]

The update is installed it seems nothing new happened i still have a problem even with the new update the problems, seems to be on the adapters when i uncheck the Mchap option my data on the vpn settings are erased this started after the the new update and it s still happening . Already tried on 3 different pcs still getting the same result i cant use Mchap on cause meraki will not work .... Anyone help am i doing something wrong ? 

[eyre-jr]

Sounds like it... the only protocol selected should be PAP - see the configuration documentation

To be clear, if you're running Windows 10 or 11, you're also going to need to make sure the optional update KB5010793 (W10) or KB5010795 (W11) has been applied if the regular Jan 11th 2022  patches are installed before you'll see client VPNs establish and maintain a connection; otherwise you'll see the 789 error (The L2TP connection attempt failed because the security layer encountered a processing error during initial negotiations with the remote computer)

[360IT]

Ensure you run updates for both win 10 and 11 in their entirey, to the point that when you check for updates the message received is "Your all up to date". Then follow instructions here: Client VPN Overview - Cisco Meraki

That is the process I have been using and have had no issues at all. 

[MikeL2022]

I can confirm it worked for me, MX64, win10, latest win update. I do have MS-CHAP checked.

Used to connect within a sec, now a few seconds to verify.

[April_Fulton]

This is due to a windows update

[Gleep52]

On my windows 11 machine - I do not see any updates - it says I am up to date but I cannot connect to my Meraki VPN still.  There are no updates available in my "Other" section either. 

When I download the patch file from the catalog, it has a windows 10 prefix, it says it is not applicable to my system.  File name: windows10.0-kb5010795-x64_7fd6ce84756ac03585cc012568979eb08cc6d583.msu

I'm certain I've used the link to windows 11 x64 (not arm I clicked the one titled " 2022-01 Cumulative Update for Windows 11 for x64-based Systems (KB5010795)" and it still says it is not compatible or needed for my system.  My winver says 21H2 for windows 11, Build 22000.466.  What is your build after applying this update?  

[Aidan_Taub]

Same here. no updates, no optional updates. The linked file is win10 (not 11). and now I can't run wusa /uninstall KB:5009566 - its giving me 80070002 System cannot find file specified.
VPN not working, and no options

[CharlieCrackle]

Are you on windows 10 21H2. ?  I found the optional update did not appear on earlier versions.

[MarkChan]

Sounds to me you are on Windows 11 Insider Beta Channel. The version 22000.466 is updated by KB5008353. 

The out of band update released by MS is targeted for 22000.438. 

I am guessing you will have to roll back KB5008353 first to the target version before you can apply hotfix, or you will have to wait for the next official release. 

Since you are on the beta channel, using the provide feedback forum might help. 

[Aidan_Taub]

yes @Mark I am. will attempt roll back then fix and will 100% feedback to microsoft on this!

[Gleep52]

You were right - I uninstalled KB5008353 and THEN I was able to update to the new KB5010795.  However, I can still no longer VPN into Meraki.  I CAN VPN into RAS, but our Meraki MX450 still does not work - it connects and then immediately disconnects?  Event log shows Event IDs 20267 (connect) and ID 20268 (disconnect) right after each other.  

CoID={A309EEF4-0E0F-000D-88A8-0AA30F0ED801}: The user <me> successfully established a connection to Meraki using the device VPN4-1.

CoID={A309EEF4-0E0F-000D-88A8-0AA30F0ED801}: The connection to Meraki made by user <me> using device VPN4-1 was disconnected.

I am set up to the specs of the Meraki docs - I set it up for my users all the time.  Thoughts?

[MarkChan]

The events id are quite generic. Does the VPN log give you more clue?

To do so:

1. Right-click the Dialup Networking folder, and then click Properties.
2. Click the Networking tab, and then click to select the Record a log file for this connectioncheck box.

The PPP log file is C:\Windows\Ppplog.txt. It's located in the C:\Program Files\Microsoft IPSec VPN folder. 

[Gleep52]

Right-click on WHAT now?  Are you still using 95?  I haven't seen a Dialup Networking folder in forever - esp not with windows 11 and its miserable new settings system that hides all the old stuff we are familiar with.  

[MarkChan]

EDIT:

In Windows 10 or 11, the way to get to the log is as follow:

1. In elevated command prompt, Type:

netsh ras set tracing * enabled

2. Try to establish the VPN connection

3. Stop the tracing in the elevated command prompt with:

netsh ras set tracing * disabled

4. Goto the %windir%\tracing folder using CD in command prompt and run:

netsh trace convert input=RRASEtwTracing.etl output=<output filename>.txt

5. The output file may contain the logging information.

ORIGINAL:

Ha, I did started with Windows 3.1 and I did get this one wrong.

Windows 11 had taken that option away. It is all in Event Log.

Without detail log, then the problem could be everyone's guess.

Uninstall WAN Miniport (L2TP) under Device Manager and rescan would be my next guess.

What would be yours?

[Rules_2301]

I have issue 😞 

[Rules_2301]

updates Windows 10

[Rules_2301]

Event Windows

[Rules_2301]

issue

[delfuego]

Etsa misma:

pausar la actualización

wusa /uninstall /kb:5009543

reiniciar

Buena suerte!

[Rules_2301]

no longer have that updateno longer have that update

[delfuego]

Looks like Microsoft's fix is still "do it yourself". Still getting user issues. Tried to have a user self-repair last night, no luck, no update available. I at least hope MS pulled down this update.

[ShadowoftheDark]

In  our case I had to download 2 Updates, the Optional R1H2 or something and then the KB5010793. 

Its a pain in the butt and takes an awful lot of time but its a solution that works

[Itchriskentas]

i tried everything the windows update fix it is not a fix i guess ... tried on differed machines even a new ones and still the problem exist . Again the problem after the update seems to be happening on the adapters .

[asowa1]

The patch did not fix the issue for me. 😞

[MarkChan]

KB5010793 superseded by KB5009596 in Windows 10

Microsoft Update Catalog

[Itchriskentas]

i must have all this updates on the server which i run meraki and on the pcs? or a specific updates cause my pc  has them all plus kb009467 should i remove that ? cause i still have  a problem i dint update my server for over 1 and a half month i disable the updates cause of problems like this so the update that caused the problem dint affect my server .

[Mike6116]

You must  first run the windows update 10 21H2  then look for the additional  KB5010793 and install

This must be done on every machine that will use the VPN to connect to the MX Network

[TEAM-ind]

This seems to be resolved with latest February updates from Microsoft.  Applied all new updates on a couple of affected clients, and VPN still works.

[MikeL2022]

I'm also verifying from one user who can now connect after the latest MS update (could not connect previously and had not installed any of the previously suggested updates).

[OVERKILL]

Yes, I can confirm the same. For the last few weeks, just ensuring a client workstation has all the latest updates resolves the problem. 

[Rules_2301]

Good afternoon, I still have and persive the problem I attach evidence

[OVERKILL]

I know it's not a "fix", but have you tried just switching to AnyConnect? 

[Rules_2301]

¿Dónde lo cambio?

[Rules_2301]

Where do I change it?

[OVERKILL]

What MX are you running? It shows up on 16.xx release firmwares on supported devices automatically (or at least it is supposed to). You'll see it as a separate tab beside the client VPN tab on the dashboard. 

[Rules_2301]

i Have 

Up to date

Current version: MX 16.16

[OVERKILL]

OK, but what model of MX are you running? For example, under Client VPN on my MX64, I see this:

 See the AnyConnect Settings tab? 

[Rules_2301]

[Rules_2301]

[OVERKILL]

Perfect, so go into AnyConnect Settings and setup AnyConnect, then download and use that client instead. That prevents you from having to deal with Microsoft breaking things with updates, as you use the Cisco AnyConnect client instead. 

[Rules_2301]

oh Yeah, thanks. 

[Mike6116]

You need to have the windows update 21Hh2 and  2 optional updates that come after it,  then you need to reconfigure tue vpn connection  and there you go....  it will connect without any issue...

[GiacomoS]

Hey team,

For those still encountering VPN issues with the Windows client, I would recommend always checking the VPN Adapter settings, as described in this KB . Microsoft has a terrible habit of resetting those checkboxes almost every Patch Tuesday, although if you are leveraging GPOs in your Active Directory environment to push the settings you may be OK.

Just to avoid any shattered dreams, please remember that the use of Anyconnect requires you to have a valid license (as per this KB). 

If you don't have a license yet, please get in touch with your partner of choice and your Cisco Meraki sales representative and they'll be happy to assist.  😁

Have a great day!

Giac