Windows laptops Losing Wireless Connection Firmware 25.9?

[RBederian]

We just recently upgraded our Access points to the latest stable firmware. Previously we were on 24.11 and did not have any issues. 

Issue we are currently seeing is affecting our Windows 7 and 10 clients. All of Mac OS X have not experienced this issue. 

The client is connected to SSID using pre-shared. After the client roams user loses wireless connection. Laptop is no longer able to connect to the SSID. Also, we have seen a few cases were the laptop is no longer able to see available SSID's. After we restart the laptop the user is able to connect to SSID. Laptop will likely experience the same issue the next day. 

Have anyone you guys seen this same issue with 25.9 firmware? I would like to troubleshoot this issue before I decide to roll back the firmware. 

[PhilipDAth]

Move to the stable release candidate 25.10.  It has this fix:

"APs in repeater mode erroneously broadcasted 802.11r support causing client connectivity issues during roaming.".

We are using 25.10 without any issues.

[RBederian]

None of my AP's are in repeater mode. Also, 802.11r was disabled on this SSID and I am not mistaken Windows does not use 802.11r when using pre-shared WPA2. It would have to be configured with Radius Auth 802.11x

[MRCUR]

We're seeing this as well unfortunately. The issue is definitely only present on Windows, and perhaps more specifically with Intel wireless cards. Macs, iOS devices and Chromebooks are not exhibiting the issue. 

This is happening on SSID's with 802.1x enabled with our own RADIUS servers. We're also seeing the devices unable to even see the SSID for a period of time. Updating the Intel driver has not changed anything and both Win 7 & 10 are exhibiting the issue. When this is happening, the AP logs "Client has left AP" numerous times every few seconds. Often times the event log will also show "XX events were not logged". Eventually the client can successfully connect, but the issue reoccurs eventually. 

I have case 02459630 open regarding this issue but haven't really gotten anywhere yet besides moving a few AP's to a test network. We'll likely need to roll back firmware very soon as the number of devices with the issue is growing. 

[RBederian]

What's strange we are having this issue with clients using pre-shared key and not radius. I would assume its related to both authenticate. 

You mentioned moving AP's to a test network. Can you elaborate what you did here? 

[MRCUR]

@RBederian So far the test network is an identical config to our production template which is applied to all other MR networks. The engineer assigned to the case requested this so we can tweak settings on specific AP's to identify a root cause. 

I strongly suspect at this point the issue is firmware based however. We'll see if we get any useful data from the AP's in the test network in the next day or two, but if not I'll need to roll the firmware back as this is a major issue for us. 

[RBederian]

I agree. Definitely an windows driver issue with Meraki firmware. The Radius Network SSID you're are seeing the issue do you have 2.4 and 5GHZ enabled or just 5GHz?

My case number is 02459891

[MRCUR]

Both 2.4 & 5Ghz are on but with band steering enabled.

[RBederian]

We rolled back to firmware 24.12. Will let you know how it goes this week. Keep me posted if you guys were able to resolve the issue.

Thanks!

[MRCUR]

Have you had any issues reported again since rolling back?

I have my primary template scheduled to roll back to 25.8 (the version it was previously running before moving to 25.9) on Friday night but have kept a few AP's in a separate network where the issue is occurring. The problem at this point is that I cannot manually reproduce the issue to get a monitor mode packet cap that support wants to see. 

[RBederian]

We haven't had any reports yet with the issue rolling back to 24.12. Interesting so rolling to 25.8 you did not see this issue. 

[MRCUR]

I haven't yet rolled back to 25.8, but it is the previous version that the template was set to and we were not seeing the issue until the 25.9 update. 

I did notice in the 25.9 release notes that this was added (not mentioned in 25.8):

• A corner case would cause AP to stop serving clients. (MR42/52/53/84)

[RBederian]

We have MR34 and MR42. We did not see the issue with the MR34. Thanks for confirming you were not able to see this issue on the 25.8 firmware. We might just wait till this issue has been resolved before updating. 

[Krewal]

Hi all. Unfortunately, we experienced the same problem on our end. Just to give you an idea from the different topic - you can track on your firewall enormous amount of the connections for the devices which eventually cause drops (firewalls have the upper limit of simultaneous connections). In our case, it is SonicWall.

Also in addition - yes, it was affecting only Windows devices, especially where we have a higher density of the APs(all our devices are MR42). We are Win7 shop so cannot say anything about Windows 10. I can also confirm this could be Intel related. We are using Latitude E7470 with Intel Dualband 8260 card. Different driver / bios versions did not make any difference. Seems like some of the phones were affected too. No problems with Mac devices. 

In our case, we decided to back to previous firmware. 3 days up as of now without any big problems. 

Looking forward to seeing Meraki applying fix. We might be in trouble in case there is a major security update. 

[MRCUR]

I am also only seeing the issue on MR42's so far and have not had any reports from buildings with MR32's. 

[MRCUR]

@Krewal & @RBederian What is the minimum bitrate set to on the SSID's you had issues with? 

[RBederian]

We have the minimum bit rate set at 18. We have also seen the issues with Windows 7. Laptops we are using are Dell 7470/7480 as well. 

It mentions as a bug fix, I would assume this has not been resolved?

• A corner case would cause AP to stop serving clients. (MR42/52/53/84)

I'm also pretty confident MR34 are not seeing this issue. 

Have one of you guys heard back from Meraki in regards to your case?

[PhilipDAth]

>We have the minimum bit rate set at 18. 

I have seen machines refuse to connect with this setting.  Make the minimum 12Mb/s for safety.

[RBederian]

What was the firmware you guys had before updating? Would like to confirm you guys were not seeing this issue with 25.8?

Was you guys bit rate higher than 12?

[MRCUR]

---

@RBederianwrote:

What was the firmware you guys had before updating? Would like to confirm you guys were not seeing this issue with 25.8?

Was you guys bit rate higher than 12?

---

We were running 25.8 prior to 25.9. 

Our min. bitrate is much higher than 12. I don't think that's related to the root issue, I just asked because a support engineer mentioned it to me. 

[MRCUR]

I moved all but three AP's back to 25.8 on Friday. We'll see if the reports of issues go away Monday/Tuesday as I expect they will. 

[RBederian]

Please keep me posted. Im currently on 24.12 and would to at least update to 25.8

[MRCUR]

So far I haven't had any reports of issues since rolling back to 25.8. I've moved a set of AP's that are known to show the issue to 25.10 at the suggestion of MR product management to see if the issue is corrected. 

[RBederian]

Same here. We will be putting an AP on 25.10 as well and test. Will follow up if we noticed the same issue

[RRuff1]

How did you get access to 24.12?  That is what I was previously using and only upgraded because the AP's were randomly rebooting.  I would gladly go back to that issue at this point

[RBederian]

you would need to reach out to Meraki support to downgrade firmware 

[Krewal]

Just talk to Meraki to roll back the firmware to previous stable for you.

[PhilipDAth]

Depending on how long ago you did this you can roll back the firmware upgrade yourself.  Go:

Organization/Firmware Upgrades

Click on the blue circle to roll back the change.

[RBederian]

Meraki mentioned this issue might of been addressed with the 25.10. I currently have one access point with this firmware and has not seen this issue. Have anyone of you guys updated to 25.10?

[PhilipDAth]

I haven't had any issues with 25.9 or 25.10.  However I updated a bunch of access points to 25.10 a couple of weeks ago after reading about other people having problems.  I'm still not having any issues.

[MRCUR]

I'm moving everything to 25.10 today after a successful test on four AP's. 

[RBederian]

Please keep me posted on how 25.10 goes

[PeterL]

I am having a semi-similar issue which may indicate that I need to do the firmware upgrade to 25.10 as well.

Started at a new company where we are having an MTU issue where I can't have a packet size greater than 1462 over wireless. The easiest way to replicate this is using ping. If I use windows with a  "ping 8.8.8.8 -l 1420 -f" (as windows take the 42 bytes for the ICMP packet) and I get a response, but 1421 and I don't.

After various tests I have isolated the issue to the wireless network as I don't get this issue over the Switch -> Router -> Internet and don't have this issue on other wireless networks so have ruled out my laptop.

Since the firmware was upgraded over 2 weeks ago I can't roll back but there has been various complaints of certain sites being unavailable and VPN connections not working "for the last month or so" which to me sounds rather coincidental.

Planning to take a non-Meraki AP onsite tomorrow and have that option to test, plus upgrade to 25.10 when I have a failback position with a secondary AP as the site principally uses wireless for all devices and there is only two APs.

[RRuff1]

I updated to 25.10 and cannot speak to the SSID issue, as I rolled back within a couple days because  throughput was horrible.  I am pending a rollback to 24.12 right now. 

[MRCUR]

@RBederian I'm rolling everything back to 25.8 tonight. Issue has not been fixed on 25.10. 

[RBederian]

Has anyone else seeing this issue with 25.10? 

[MRCUR]

I'm continuing to see this issue in 25.8 although much less often than 25.9/25.10. 

[PeterL]

Is there a way to roll back to 25.8 if it has been over 2 weeks as the Meraki portal doesn't allow rollback if it has been longer than 2 weeks?

Is there a way to force the downgrade as I am not familiar with the Meraki product suite and it isn't clear there is a way to do it?

[RBederian]

You would need to reach out to Meraki support

[RRuff1]

Open  a case with Meraki and request a rollback

[PeterL]

Cheers, I spent an hour with Meraki support explaining it and they are off to test in the lab. Asked if I could get access to roll back but the engineer was very hesitant and said the latest was the best version to be on. So going to wait and see if he can duplicate the issue in their lab. All the tcpdumps in the world showing the issue only existing on the APs and not passing to the switches couldn't convince him to let me try rolling back the firmware.

[RBederian]

Curious which access point models are you seeing this issue?

[PeterL]

We are using MR32s two APs in a single office running around 30 people. The APs themselves are 10 Meters apart and have a fairly even split of clients connected. It's completely Meraki here, MX84 -> MS220-24P -> APs plus a number of wired devices. So it really couldn't get any simpler.
Plugging in a Cambium E500 in sorted the issue and I can clearly see that the packets over 1462 are received by the AP, but never forwarded to the switch when I did a tcpdump on the uplink port to the MX84.

[RRuff1]

I am experiencing this issue on Meraki MR 52's.  We have 35 plus deployed across 3 floors. 

[MRCUR]

---

@RRuff1wrote:

I am experiencing this issue on Meraki MR 52's.  We have 35 plus deployed across 3 floors. 

---

Mention this thread/case numbers to the assigned engineer so they can get everything together if your issue is related. The more info the engineering team can gather the better. 

[PeterL]

I just received a response back from support. I find it hard to believe but wanted to check if it is true as my initial thoughts were "you can't be serious"

I tried reproducing the environment again and I found that L3 roaming reduces the MTU from 1500 to 1448 and I confirmed this with Engineering aswell.
So the reason why we were getting that error in your tests is because of the L3 Roaming enabled. If you change the L3 Roaming to make a part of the LAN you would be able to send pings without any issues. Let me know if there is a specific requirement for which you need it to be in L3 roaming mode.

But the whole point of L3 Roaming is to have seamless handover between APs. Or do I need to change over to bridging mode to get this working.

I've not seen any mention in any documents and the only place I have found that talks about the MTU limitation is on a Cisco forum: https://supportforums.cisco.com/t5/lan-switching-and-routing/3560x-mtus-and-icmp-unreachables/td-p/2...

The issue is that packets are silently discarded with no notification back to the wireless client and never sent to the destination.

I'm surprised this hasn't been noticed before.

[MRCUR]

@PeterL L3 Roaming is only needed if your network is architected in a way that a user would be roaming among AP's which cannot bridge them to the same VLAN. I do agree that you've found a bug which shouldn't be occurring with L3 roaming enabled, but I would highly recommend moving to a more traditional deployment where you bridge clients to the LAN as I think you'll have a much better experience overall. 

See here for the official L3 roaming documentation that explains the use cases where it's recommended: https://documentation.meraki.com/MR/Client_Addressing_and_Bridging/Layer_3_Roaming

I do not believe the issue you're seeing is related to what's being discussed in this thread regarding issues with MR 25.9 firmware in Intel WiFi adapters. 

[RRuff1]

We are having this issue across all SSID's both using preshare keys, and Radius authentication.  In addition, we are experiencing extremely slow throughput causing a host of complaints across our entire enterprise.  In an attempt to fix the issue, I upgraded to 25.10 as it is a "stable release candidate".  The throughput seems to be as bad, if not worse.and the SSID issue is not as frequent but still there.

[Leejohn1]

What is the recommendation for this? 🙂

[MRCUR]

---

@Leejohn1 wrote:

What is the recommendation for this? 🙂

---

Roll back your firmware to the last known good version.