Wireless - Invalid MIC - EAPoL 4-way handshake is failling

[RaphaelL]

Hi ,

Any wireless Guru that could help me troubleshoot/understand this issue that we are having ? 🤔

Since we upgraded to MR29 ( I tried ANY versions of MR29 ) , users are randomly getting errors ( 20-25s of packetloss ) on reassoc. 

SSID is WPA2-Enterprise with ISE. 802.11r is enabled , CoA disabled and 802.11w disabled.

Dashboard always shows theses logs : 

Packet captures almost always show that the 4-WAY EAPoL is missing Message 3-4  :

All our workstations are using Intel wireless NIC. We are running 22.170.3 but I have tried other version such as the latest 22.200.2. Same result. 

Downgrading to MR28 OR disabling 802.11r solves the issue.

Any tips / ideas ?

[alemabrahao]

Probably it's a client incompatibility. Take a look at this.

https://www.slideshare.net/akg_hbti/80211r-enhanced

[RaphaelL]

Not impossible ! But I mean HP laptops and Intel AX cards are fairly common. 

I have a hard time feeling that we are the only one experiencing this issue 😕

Disabling FT solves the issue , but the MIC errors ( 4-way EAPoL HS)  are not related to FT , so I'm a little confused to where should I put my efforts.  

And yes I do have a support case open.

[KarstenI]

The missing frames 3 and 4 are the logical consequence of the MIC failure. If the MIC can’t be verified after frame 2 the AP has to abandon the session. But this doesn’t explain why this is happening …

[RaphaelL]

Exactly. Is the client sending a bad MIC ? or is the AP 'calculating' the bad MIC 🤔

[KarstenI]

I would bet on a bug in the calculation of the keys on the AP. Disabling .11r changes the calculation on both AP and client, but the difference in firmware versions should only change the AP behavior.

[KarstenI]

Another strange thing is the TLS exchange before the 4 way handshake. This means it is not happening when doing a fast roam but instead on the initial connection?

[RaphaelL]

The EAP-TLS packets shown are from the client that succeded the 4WAY HS. They are not related to the one that failed.  Sorry for that confusion

[NitinVats]

We are experiencing the same issue with our Meraki deployment ever since we enabled 802.11r. We enabled it to overcome the 802.1X EAP-TLS auth delay issues. We also upgraded the drivers to latest version. Are you able to find a solution to this yet other than disabling 802.11r?

[alemabrahao]

@NitinVats , while many customers enable 802.11r within their network without issue, some legacy devices may not connect to an 802.11r network. 802.11r is a recommended feature due to its many benefits, however a device audit is encouraged first to ensure that mission critical devices are not affected.

[RaphaelL]

Nope , not fixed yet. Still taking packet captures and debugs with Support. 

What wireless NICs do you have ?

[NitinVats]

@RaphaelL @alemabrahao We have received same errors of "EAPOL Invalid MIC" from supplicants with drivers of version 20.70.25.2, 20.70.32.1 (AC 8265), 22.170.0.3 (AX201). All are HP surface pro laptops. We enabled Fast roaming on advise of Meraki TAC and made sure our supplicants support 802.11r. 

Now Meraki TAC is putting this issue on supplicant behavior and advising drivers update. We did that and now this issue "EAPOL Invalid MIC" that causes 5-10 secs of auth delay during initial EAPOL 4 way handshake

I see HP had given a solution in their Knowledge Article to Update Wi-Fi drivers but what if issue still appears after updating drivers? No Answers to that 
Link to HP Knowledge article: https://support.hpe.com/hpesc/public/docDisplay?docId=mmr_sf-EN_US000005284

[RaphaelL]

What OS is that ? Where did you get that 'advanced settings' window ?

[NitinVats]

@RaphaelL Its a win10 Machine. Under Wi-Fi Properties>> SecurityTab>> You have to select WPA2-Enterprise>> Advanced Settings>> 802.11 settings. 
Make sure that all your supplicants have this enabled to support 802.11r.
I also got one tip to upgrade the WIN10 TCPIP Lib version from 21H2 to 22H2.

[RaphaelL]

Same here. We are on 21h2 : 

[NitinVats]

@RaphaelL So after downgrading to MR 28 does the issue resolve? Keeping 802.11r enabled on meraki.

[RaphaelL]

From my experience downgrading to MR28 fixes the issue ( with or without 802.11r ). Will be trying this with Support this week.

[NitinVats]

@RaphaelL , Please keep me posted on the results. We have also asked support to lab recreate the same with MR28 and MR29. If its really an issue with MR29 then Meraki should Open a bug for "Wrong MIC calculation during EAPOL handshake on MR29s".

[NitinVats]

@RaphaelL 

Please refer to the Microsoft Article KB5003690 which states that this is a known issue with Windows 10 feature versions 2004, 20H2, and 21H1 and Intel AX201, Intel AC7260, AC, and Qualcomm QCA61x4A. Windows 10 versions prior to 1909 function properly.

• Effectively this means that to get the fix, Windows 10 clients running feature version 2004 or later need to have the "July 2021 cumulative quality update" or later which is included in 22H2. So the solution is to Feature update WIN 10 to 22h2.
• With feature update to 22H2, windows will download a regular monthly cumulative security update without requiring reinstallation for devices already running version 21H2, 21H1, or 20H2.

[NitinVats]

Refer to the snippet from KB5003690

[RaphaelL]

Hi , 

I don't see that snippet in the link you refered.  

1- We are running Win11 and not Win10

2- 802.11W is not enabled

3- Everything works fine when downgrading to MR28 , how can this be client side ?

[NitinVats]

@RaphaelL , You have to expand the Improvement and Fixes Tab to see those points in the link I provided. 
Downgrading to MR28 fixes everything but that firmware will be soon end of support also has lot of other issues associated with it. Any of your AP's upgraded to MR30 firmware to see if that issue  is still there?
Also please advise, how have you taken the captures you shown in your first post that shows Message 3,4 missing. Is it a wireshark on client side or captures on Meraki AP for a particular client or Monitor mode captures?

[RaphaelL]

Support has not suggested to upgrade to MR30 since there are no fixes associated to my issues.

The pcap was taken on a RPI in monitor mode since Intel wireless cards do not support monitor mode.

[NitinVats]

@RaphaelL, That's the issue on our end we need to use macbooks for monitor mode which doesn't experience this issue. Getting Raspberry-Pi wi-fi sniffer would be a difficult task.

Also please advise, How is MR28 available to you for rollback? Meraki has removed it already from the stable releases. I assume you might got it rolled back to MR28 through TAC?

[RaphaelL]

Setting the RPI with a USB dongle wasn't too hard.  

Correct , support had to rollback to MR28

[Melchi]

Hi @RaphaelL,

Do you know if you were experiencing MIC failures in an 802.11r SSID while having PMK caching disabled on the Windows clients?

Could you please check in your monitor mode captures if the MIC failures occur when STAs include two PMKIDs in their message 2? You can check it on EAPOL Message 2 under 802.1X Auth > WPA Key data > RSN info > PMKID Count/List

Lastly, if you have an open case with Support, could you please provide a working capture to Meraki Support showcasing multiple successful roams in r28 for comparison?

Sorry for the inconvenience, and thanks!

[RaphaelL]

Hi @Melchi  ,

1- PMK caching wasn't disabled at any moment during the tests on either MR28 or MR29

2- 

This is the EAPoL Message 2 , right after that the AP responded with a deauth

3- Yes this is what I'm curently trying to achieve 🙂 

Thanks for your help !

[RaphaelL]

I'm now 1000% lost. Same issue with MR28 : 

It seems that once the network is upgraded to MR29 , doesn't matter if you rollback to MR28 , you are stuck with it. 

I have over 500 networks and most of them have 0 'EAPoL invalid MIC' logs. But another network was upgraded to MR29 and then rollbacked to MR28 shows the same behavior.  I can see very few logs on MR28 networks with the same error. 

EDIT : The M2 that is failing ALWAYS contains 2 PMKID. Then the client only sends 1 PMKID and it works... 

That's curious..

[RaphaelL]

Update !! 

Hi,

Yes, they were able to reproduce it in-house. Our understanding so far is that the intel NIC seems to be sending 2 PMKID values to the AP during the re-auth period, since the AP is only expecting one, it leads to the AP sending a de-auth to the client.

I'm guessing that @Melchi  is either a Wifi guru and/or is working actively on my case 🙂 He was spot on !

[KarstenI]

Quite strange to have this behavior as multiple PMKIDs are allowed since “forever” … Hopefully it gets solved soon.

[RaphaelL]

To be honest , my wireless knowledge is pretty limited. Learning everyday , I assumed that multiple PMKIDs are a problem. Well in my case everytime there's 2 , the AP sends a de-auth. Is the client sending the 'correct' PMKIDs ? Good question

[Melchi]

Per 802.11 Standard, section 13.8.2 FT Auth sequence: contents of first message:

If present, the RSNE shall be set as follows:

Version field shall be set to 1.

PMKID Count field shall be set to 1.

PMKID List field shall contain the PMKROName.

Multiple PMKIDs are allowed in certain situations. However, if I am understanding the standard correctly, this does not seem to be one though, and the supplicant is passing two PMKIDs.

Have you seen this issue with any device that is not running Windows? What about Windows with a non-Intel driver?

[KarstenI]

Very good point. On my previous answer I didn't think about that in this scenario only the FT sequence is relevant where indeed only one PMKID is allowed. Important here is that in IEEE wording "shall" is equivalent to an RFC "must" and not to an RFC "should".

[RaphaelL]

Hi Melchi ,

All our machines are from HP and they all are using Intel NICs. I had confirmed in the past that AX200 and AX201 were having the same issues.

We also have BYOD devices on that same SSID eg : Samsung , Iphones and so on. I wasn't have to confirm if the issue is present , but I suspect that it is working fine.

I will be testing next week with the same laptop but running Win 22H2 instead of 21H2.

[RaphaelL]

Just installed a USB wireless NIC : 

Will be trying that for a couple days.

[NitinVats]

@RaphaelL Have you tried testing the same with having PMK Caching disabled on client side? Would fast roaming still work if we have that disabled is the next question.

Since you are on win11, whats the TCP IP Lib version you are using and the network driver version for AX201s?

[RaphaelL]

No haven't tried PMK caching disabled yet.

Don't know how to view the TCP IP lib version 😞  and the drivers is 22.170.3 but I have tested the latest 22.200 without success.

[NitinVats]

@RaphaelL , To view TCP/IP Lib version you can go to My Pc >>Properties >> windows specifications. 

I believe you should try with having PMK Caching disabled on the end clients under 802.11 advanced settings as Fast roaming would still work if we have that disabled. Ideally its a supplicant issue which has to be fixed by microsoft, is that a case raised with microsoft yet?

[RaphaelL]

We haven't raised a ticket with Microsoft since we had 0 clue where the issue was. On the client or on the AP. But we will probably open one !

[jpavonm]

I've experienced the same issues during the past 2 weeks using Cisco WLAN Infrastructure and Catalyst 9800.

After discussing this with Cisco BU the engineer told me they were investigating this a month ago. They reached to a conclusion that the problem was that Windows clients were tunring from using OKC to legacy SKC, and that's why the client send a different PMKID to the AP. As the PMKID shared by Windows is not the one that the controller has and share for all APs in the same group, it rejects the client and the client is stuck with no current association. After 3 minutes Windows reset the connection an connect again with a full authentication so a new PMKID is generated until it fails again when roaming.

This happen when using non-centralized forwarding such in the Meraki case.

Cisco has released a patch for this behaviour to full de-authenticate the Windows client if the controller sees invalid PMKID, but they encourage clients to open a support case with Microsoft to analyse this and solve it.

[NitinVats]

Hi @jpavonm , what was the patch released for Cisco Meraki firmware to deauth the client when it sends invalid PmK Id?

[AllenWu]

Hi @jpavonm I am experiencing the similar issue on 9800, could you tell me what the Patch is ?

[jpavonm]

Cisco has released an AP Service Pack to address this on 17.9.3 code (APSP1). This patch is integrated on 17.9.4 and next to be released 17.9.5, but also in 17.12.1 and 17.12.2

[jpavonm]

@NitinVats the patch is for Cisco WLAN Infrastructure not for Meraki.

They are releasing that patch for all software codes but for Cisco software not MEraki one, unless Meraki will be doing it as well.

[jpavonm]

But I see some omprovement has been added into release 30.1:

[RaphaelL]

Edit :  I made a HUGE mistake. 

I said I tested the latest Intel AX drivers. This is false. I only tested 22.200.2 but 22.220.0 is out and has a fix :

When connected to certain wireless APs in 802.11ac or 802.11ax mode, network
connectivity loss (Windows System Event ID 5002) might occur after roaming.

I also tested a Asus USB wireless NIC and it was 100% working fine. So I'm 99.99% sure those issues were drivers related ( again ). So hard to spot these issues.

[RaphaelL]

Version 22.220.0.4 is still sending 2 PMK . Same errors

Ughhhhhhhhh this is killing me.

[NitinVats]

@RaphaelL Have you heard anything on this from Microsoft support yet? This looks to be a bug with the registry value of the PMK Cache timer which is not working as expected for the windows supplicants, hence sending two PMK Values at different times.

[RaphaelL]

Yeah we are waiting on a special KB from Microsoft. That KB will be deployed next year.

Still trying to test it out to find out if it fixes our problems or not

[ChristophJ]

Hey RaphaelL,

I saw that Intel released a new driver in June (22.230.0.8) which also says they resolved an issue with clients not able to connect to specific wireless APs. So might be good to check it out:

https://www.intel.com/content/www/us/en/download/19351/windows-10-and-windows-11-wi-fi-drivers-for-i...

Just installed it and will test it out on my Dell Precision 5550 with an AX201 card in the next days. Not sure if I can get a capture. The issue happens super randomly for me...

[RaphaelL]

Were you having the EAPoL errors ? 

What I was experiencing is a bug from Microsoft that allows the client to send 2 PMK. Somewhere around MR29 , Meraki stopped ignoring the fact that some clients were sending 2 PMK. 

I might try that driver , but I doubt it will fix my issues.

[jpavonm]

@RaphaelL do you have any information about that special KB or the defect ID they have associated to this issue?

Thanks a lot

[RaphaelL]

Not at the moment ! Limited info received about that part. Will let you know once I have it.

[NitinVats]

Can you please share the support reference case for Microsoft which you have opened! So I can use it as a reference case.

We have also recently opened one case #2306190050002494 for this 2 PMK ID Issue. 

[RaphaelL]

Will do. I don't have the # yet.  Were you able to get a hand on that KB ?

[Raghu_Kuri]

We are battling with these issues as well. The main concern is our clients being on Teams calls and observing a video freeze or an audio issue. This happens after few seconds of roaming to a different place on the same floor or sometimes even when the client is static at one spot on the floor but 802.11k,v pushing the client to roam.

We are using 802.1x authentication with EAP-TLS. Currently running on Beta 30.1 on Meraki and with our Windows 10 clients and drivers running on 220.2. 802.11 K, V, R all enabled on Meraki side.

The timeline on the health of dashboard shows up EAPOL timeout messages during all roaming situations. We rather dont expect any sort of EAPOL messages while roaming with 802.11R.

The first feeling was our CoA is interacting with 802.11R and we disabled CoA.. However we continued seeing EAPOL timeouts. We believe the 802.11R is still not happening. Have been researching all the day along with our WPS tech-savvy and Our next step would be to enable PMK caching and pre-authentication on the windows clients (which apparently we found was missing) via Intune. Meraki TAC is continuing their investigation and I will tomorrow post them live captures during roaming & a time out.

[RaphaelL]

Interesting.  Could you share a screenshot of those missing configurations ?

I though that K,V,R wasn't configurable with Intel NICs as they were all enabled by default.

[ChristophJ]

I ran into the same issue with the Windows clients in my environment. Enabling PMK Caching in the Intune profile and deploying it fixed it. So roaming works  now.
You don't need to activate Pre-Authentication. The Meraki APs are not able to do pre-auth so having it enabled made no difference for me.

What I can see is that I run into the "Invalid MIC" error now when the clients roams which looks like the error RaphaelL runs into. It happens less often than the EAPoL timeout for me, but it happens.

Looks like a issue with Windows (running Win11 22H2), with the Macbooks we have I can't see the MIC errors happening.

[Raghu_Kuri]

Thanks ChristophJ. Can you share more information on why Meraki AP's are unable to do Pre-Auth. I couldnt see any reference documents. I see a value in having the clients pre-auth with close by AP's and hence went for it. I dont see a reason why they do not do pre-auth.

[ChristophJ]

Hi Raghu,

the beacon frames sent by the APs indicate that it's not pre-auth capable:

That's why I didn't enable it in my case.

[Raghu_Kuri]

Thanks Christoph, I appreciate the screenshot. Just checked even MR46's that we use do not support Pre-Auth. Its a pity that its not supported by Meraki.

[ChristophJ]

Pre-Auth is a fairly old feature that predates 802.1r and even OKC.

If you enabled Key Caching on Windows and 802.1r on Meraki that should already help with roaming by a lot.

[Raghu_Kuri]

AX200 802.11r windows 10 PMK Caching - Intel Community

[Raghu_Kuri]

If you are pushing the profile via Intune, just scroll down on the Wifi settings for the profile in Intune and the settings for PMK caching will be found. 

[Raghu_Kuri]

My windows 10 has the new profile we pushed that contains PMKID and Pre-Auth enabled. The good news is I see no EAPOL timeouts anymore. We are going to have more pilot users added into this group today and will monitor how it goes for them this week and half of next week before we push this to all our staff. Also to let you know Intel has today released the 230 driver. I will just get this onto my windows and monitor if the stability is maintained.

[decassuncao]

Try disable .r

[NitinVats]

@RaphaelL Anything yet from Microsoft on the KB articles for the 2 PMK ID Issue? 

[RaphaelL]

Not yet. I'm on vacation. I will have to check next monday

[Jager01]

Did you ever manage to find a solution for this?  What exactly did Microsoft tell you about the special KB that will contain a fix? 

[RaphaelL]

Still no recent news ! Last I heard they might deploy that KB in 2023. I would expect nov/dec. I will keep that thread updated !

[Jager01]

Thanks for your reply! 

Hopefully they will release this KB soon. Is there any written confirmation from Microsoft about this problem? I cannot find anything about it. 

[RaphaelL]

Hi everyone.

No news/updates from Microsoft , but I saw this today :

Going to test this next week and keep everyone informed. 

[RaphaelL]

Update 2023-10-10 

It seems that the double PMK issue is only affecting us as per Microsoft : 

• The fix will be publicly available in April via what I believe will be called 24H1 (build 2304)
• We only have XXXXXX ( our company ) raising this scenario to us at this time.  No other customers have opened ticket with us on this same scenario.

So we are stuck with MR28.6 until atleast April 2024. yay.

[jpavonm]

@RaphaelL do you have a bug id, or ticket number, to which we can reference to Microsoft?

In my case, the problem is that WinTel team do not open ticket to MS unless this ould be something critical, which is not the case.

[RaphaelL]

Our case is 2306080040008631. I don't have the bug ID or anything else at the moment.

[RaphaelL]

Hi guys , 

I know this is a long post, but I really need everyone that has the same issue as us ( Client sending 2 PMK and the APs responds with a de-auth ) to open cases with Microsoft and post your case number here.  We need to push them to release the Windows KB sooner than expected. 

Cheers !

[NolanHerring]

So I'm just now starting out testing with our Intune Windows 11 (23.30.0 ax211) and iPad is able to do roaming with 802.11r just fine. I also have 802.11w enabled, since iPad and these Dell laptops both say they support 802.11r and 802.11w, so figured why not. 

The Windows laptop was having hard disconnect, so basically no roaming when I was testing. Logs showing EAPoL timeout and Invalid PMKID. Using EAP-TLS local auth option.

I just discovered though that the wireless Intune profile doesn't have PMK Caching enabled, so getting that fixed and will test again. Hoping that was the issue.

Disabled 802.11r and 802.11w and client roams fine (not fast roaming, but better than a no-roam scenario lol). Will need to test with 802.11w on/off as well to see if that is causing any issues, since I saw it mentioned above in this thread.

[RaphaelL]

Bummer !  Yeah it looks like the same issue that we are having with Windows and Intel NICs. 

Microsoft has delayed the fix 2 or 3 times. ETA April at the moment OR like you mentionned , disable FT for the moment which is better than no roaming at all

[NolanHerring]

I forgot to mention I'm on MR44 with 30.6 code.

Yeah I'm honestly really hoping it was just the stupid PMK Caching being set to NO on the Intune profile. Otherwise I suppose I could at least use 802.11r Adaptive (instead of Enabled) since that is suppose to only use 802.11r for Apple devices, but if its anything like their OS detection then maybe not lol. That is hit or miss. 

[NolanHerring]

Ugh, double correction here. My actual testing was at my office which is still 28.7 MR44. The new site is 30.6 with MR44, but I haven't been able to test that yet in person, but I would suspect the same results.

[RaphaelL]

One year later ... we have received the test KB from Microsoft and I'm currently testing 802.11r and roaming. 

From 2 days of testing this looks promising ! No issues so far.🤞

[jpavonm]

That's good news @RaphaelL , could you please share the KB Id that seem to dfix these issues please?

[RaphaelL]

Hi !  Sadly it is a private KB and cannot be shared ! The public release is planned next month or so 

[KrissR]

Hi Raphael, I just wanted to thank the hell out of you. Et considérant que tu es du Québec, merci infiniment!

We just got a new MR36+MR46 Setup on FW30.6 to replace our old Ubiquiti stuff and I've been pulling my hair trying to diagnose this related issue for the last 2 weeks. All Windows 10 + Intel AX201/AC9560

Do you have any more problems with the new Windows KB by the way? Is everything still good?

[RaphaelL]

Ça me fait plaisir ! 

Sad news at the moment , the KB is not working at the moment. Currently troubleshooting with MSFT but in the mean time I'm also testing a "special" MR build that allows 2 PMK from a client. Downside is that it breaks WPA3 ( which we don't use atm ) 

[jpavonm]

This is really bad news as we use WPA3 here. This regression defects are crazy and the worst is that MS does not inform about them on the release notes nor they have a bug track tool to look for all of the current ones.

So we need to expect additional issues with wireless connectivity in the future from MS side as far as they release this KB, or are you reporting them back this so to fix it?

[RaphaelL]

Still testing that KB. They have requested a bunch of logs on my end to analyze. They re-pushed the public release to june.

[source]

I'm also interesting in this KB, because we have the same issue at the moment.

[rc23]

Also interested - same issues here. 

[rc23]

@RaphaelL I see in June 2024's patch cycle they are including something regarding wireless, but it doesn't appear to me to be directly related to this bug. It appears to be more of a security thing - CVE-2024-30078 .

Microsoft June 2024 Security Updates - Microsoft Community

Have you heard anything else about when they release this? Did they ever give you a bug id? I am working on our process to contact Microsoft support and plan to open my own case. 

[RaphaelL]

Hi ,

I don't think that this fix is going to fix the issue about PMKs. 

Last time I heard from MS , they were trying a new KB with another client which could repro the issue on demand. 

We are still waiting for a permanant fix on win 21h2++ 

[rc23]

Agreed - thanks for the reply. I will update this thread if I get any additional information on my side.

[patricpa]

Hi,
I had a similar issue with DELL Optiplex stations running Windows 10: the DELL station was reusing the same PMKID when roaming from one AP to another AP , or when switching between the 5 GHz and 2.4 GHz spectrums on the same AP.
This behavior does not meet the 802.11 standards and was causing client disconnections and reconnections for a few seconds with incident in APs logs Invalid PMKID and the client was disconnected (correct behavior is : AP#1 PMKID #1, AP#2 PMKID#2, if client roams back from AP# 2 to AP# 1 it can reuse the PMK#1,  then from AP#1 to AP#2 it can reuse PMKID#2 )

After upgrading the same station to Windows 11, the issue was resolved. The PMKIDs were different while roaming, as expected.

Hope this helps.