Windows laptops Losing Wireless Connection Firmware 25.9?

RBederian
Here to help

Windows laptops Losing Wireless Connection Firmware 25.9?

We just recently upgraded our Access points to the latest stable firmware. Previously we were on 24.11 and did not have any issues. 

 

Issue we are currently seeing is affecting our Windows 7 and 10 clients. All of Mac OS X have not experienced this issue. 

 

The client is connected to SSID using pre-shared. After the client roams user loses wireless connection. Laptop is no longer able to connect to the SSID. Also, we have seen a few cases were the laptop is no longer able to see available SSID's. After we restart the laptop the user is able to connect to SSID. Laptop will likely experience the same issue the next day. 

 

Have anyone you guys seen this same issue with 25.9 firmware? I would like to troubleshoot this issue before I decide to roll back the firmware. 

 

95 REPLIES 95
PhilipDAth
Kind of a big deal

Move to the stable release candidate 25.10.  It has this fix:

"APs in repeater mode erroneously broadcasted 802.11r support causing client connectivity issues during roaming.".

 

We are using 25.10 without any issues.

None of my AP's are in repeater mode. Also, 802.11r was disabled on this SSID and I am not mistaken Windows does not use 802.11r when using pre-shared WPA2. It would have to be configured with Radius Auth 802.11x

MRCUR
Kind of a big deal

We're seeing this as well unfortunately. The issue is definitely only present on Windows, and perhaps more specifically with Intel wireless cards. Macs, iOS devices and Chromebooks are not exhibiting the issue. 

 

This is happening on SSID's with 802.1x enabled with our own RADIUS servers. We're also seeing the devices unable to even see the SSID for a period of time. Updating the Intel driver has not changed anything and both Win 7 & 10 are exhibiting the issue. When this is happening, the AP logs "Client has left AP" numerous times every few seconds. Often times the event log will also show "XX events were not logged". Eventually the client can successfully connect, but the issue reoccurs eventually. 

 

I have case 02459630 open regarding this issue but haven't really gotten anywhere yet besides moving a few AP's to a test network. We'll likely need to roll back firmware very soon as the number of devices with the issue is growing. 

MRCUR | CMNO #12

What's strange we are having this issue with clients using pre-shared key and not radius. I would assume its related to both authenticate. 

 

 

You mentioned moving AP's to a test network. Can you elaborate what you did here? 

MRCUR
Kind of a big deal

@RBederian So far the test network is an identical config to our production template which is applied to all other MR networks. The engineer assigned to the case requested this so we can tweak settings on specific AP's to identify a root cause. 

 

I strongly suspect at this point the issue is firmware based however. We'll see if we get any useful data from the AP's in the test network in the next day or two, but if not I'll need to roll the firmware back as this is a major issue for us. 

MRCUR | CMNO #12

I agree. Definitely an windows driver issue with Meraki firmware. The Radius Network SSID you're are seeing the issue do you have 2.4 and 5GHZ enabled or just 5GHz?

 

My case number is 02459891

MRCUR
Kind of a big deal

Both 2.4 & 5Ghz are on but with band steering enabled.

MRCUR | CMNO #12

We rolled back to firmware 24.12. Will let you know how it goes this week. Keep me posted if you guys were able to resolve the issue.

 

Thanks!

MRCUR
Kind of a big deal

Have you had any issues reported again since rolling back?

 

I have my primary template scheduled to roll back to 25.8 (the version it was previously running before moving to 25.9) on Friday night but have kept a few AP's in a separate network where the issue is occurring. The problem at this point is that I cannot manually reproduce the issue to get a monitor mode packet cap that support wants to see. 

MRCUR | CMNO #12

We haven't had any reports yet with the issue rolling back to 24.12. Interesting so rolling to 25.8 you did not see this issue. 

MRCUR
Kind of a big deal

I haven't yet rolled back to 25.8, but it is the previous version that the template was set to and we were not seeing the issue until the 25.9 update. 

 

I did notice in the 25.9 release notes that this was added (not mentioned in 25.8):

  • A corner case would cause AP to stop serving clients. (MR42/52/53/84)
MRCUR | CMNO #12

We have MR34 and MR42. We did not see the issue with the MR34. Thanks for confirming you were not able to see this issue on the 25.8 firmware. We might just wait till this issue has been resolved before updating. 

Hi all. Unfortunately, we experienced the same problem on our end. Just to give you an idea from the different topic - you can track on your firewall enormous amount of the connections for the devices which eventually cause drops (firewalls have the upper limit of simultaneous connections). In our case, it is SonicWall.

 

Also in addition - yes, it was affecting only Windows devices, especially where we have a higher density of the APs(all our devices are MR42). We are Win7 shop so cannot say anything about Windows 10. I can also confirm this could be Intel related. We are using Latitude E7470 with Intel Dualband 8260 card. Different driver / bios versions did not make any difference. Seems like some of the phones were affected too. No problems with Mac devices. 

In our case, we decided to back to previous firmware. 3 days up as of now without any big problems. 

 

Looking forward to seeing Meraki applying fix. We might be in trouble in case there is a major security update. 

 

 

MRCUR
Kind of a big deal

I am also only seeing the issue on MR42's so far and have not had any reports from buildings with MR32's. 

MRCUR | CMNO #12
MRCUR
Kind of a big deal

@Krewal & @RBederian What is the minimum bitrate set to on the SSID's you had issues with? 

MRCUR | CMNO #12

We have the minimum bit rate set at 18. We have also seen the issues with Windows 7. Laptops we are using are Dell 7470/7480 as well. 

 

It mentions as a bug fix, I would assume this has not been resolved?

  • A corner case would cause AP to stop serving clients. (MR42/52/53/84)

I'm also pretty confident MR34 are not seeing this issue. 

 

Have one of you guys heard back from Meraki in regards to your case?

>We have the minimum bit rate set at 18. 

 

I have seen machines refuse to connect with this setting.  Make the minimum 12Mb/s for safety.

What was the firmware you guys had before updating? Would like to confirm you guys were not seeing this issue with 25.8?

 

Was you guys bit rate higher than 12?

MRCUR
Kind of a big deal


@RBederianwrote:

What was the firmware you guys had before updating? Would like to confirm you guys were not seeing this issue with 25.8?

 

Was you guys bit rate higher than 12?


We were running 25.8 prior to 25.9. 

 

Our min. bitrate is much higher than 12. I don't think that's related to the root issue, I just asked because a support engineer mentioned it to me. 

MRCUR | CMNO #12
MRCUR
Kind of a big deal

I moved all but three AP's back to 25.8 on Friday. We'll see if the reports of issues go away Monday/Tuesday as I expect they will. 

MRCUR | CMNO #12

Please keep me posted. Im currently on 24.12 and would to at least update to 25.8

MRCUR
Kind of a big deal

So far I haven't had any reports of issues since rolling back to 25.8. I've moved a set of AP's that are known to show the issue to 25.10 at the suggestion of MR product management to see if the issue is corrected. 

MRCUR | CMNO #12

Same here. We will be putting an AP on 25.10 as well and test. Will follow up if we noticed the same issue

How did you get access to 24.12?  That is what I was previously using and only upgraded because the AP's were randomly rebooting.  I would gladly go back to that issue at this point

you would need to reach out to Meraki support to downgrade firmware 

Just talk to Meraki to roll back the firmware to previous stable for you.

 

 

PhilipDAth
Kind of a big deal

Depending on how long ago you did this you can roll back the firmware upgrade yourself.  Go:

Organization/Firmware Upgrades

Click on the blue circle to roll back the change.

 

Screenshot from 2018-03-24 08-29-52.png

Meraki mentioned this issue might of been addressed with the 25.10. I currently have one access point with this firmware and has not seen this issue. Have anyone of you guys updated to 25.10?

I haven't had any issues with 25.9 or 25.10.  However I updated a bunch of access points to 25.10 a couple of weeks ago after reading about other people having problems.  I'm still not having any issues.

MRCUR
Kind of a big deal

I'm moving everything to 25.10 today after a successful test on four AP's. 

MRCUR | CMNO #12

Please keep me posted on how 25.10 goes

I am having a semi-similar issue which may indicate that I need to do the firmware upgrade to 25.10 as well.

 

Started at a new company where we are having an MTU issue where I can't have a packet size greater than 1462 over wireless. The easiest way to replicate this is using ping. If I use windows with a  "ping 8.8.8.8 -l 1420 -f" (as windows take the 42 bytes for the ICMP packet) and I get a response, but 1421 and I don't.

After various tests I have isolated the issue to the wireless network as I don't get this issue over the Switch -> Router -> Internet and don't have this issue on other wireless networks so have ruled out my laptop.

 

Since the firmware was upgraded over 2 weeks ago I can't roll back but there has been various complaints of certain sites being unavailable and VPN connections not working "for the last month or so" which to me sounds rather coincidental.

 

Planning to take a non-Meraki AP onsite tomorrow and have that option to test, plus upgrade to 25.10 when I have a failback position with a secondary AP as the site principally uses wireless for all devices and there is only two APs.

I updated to 25.10 and cannot speak to the SSID issue, as I rolled back within a couple days because  throughput was horrible.  I am pending a rollback to 24.12 right now. 

MRCUR
Kind of a big deal

@RBederian I'm rolling everything back to 25.8 tonight. Issue has not been fixed on 25.10. 

MRCUR | CMNO #12

Has anyone else seeing this issue with 25.10? 

MRCUR
Kind of a big deal

I'm continuing to see this issue in 25.8 although much less often than 25.9/25.10. 

MRCUR | CMNO #12

Is there a way to roll back to 25.8 if it has been over 2 weeks as the Meraki portal doesn't allow rollback if it has been longer than 2 weeks?

Is there a way to force the downgrade as I am not familiar with the Meraki product suite and it isn't clear there is a way to do it?

You would need to reach out to Meraki support

Open  a case with Meraki and request a rollback

Cheers, I spent an hour with Meraki support explaining it and they are off to test in the lab. Asked if I could get access to roll back but the engineer was very hesitant and said the latest was the best version to be on. So going to wait and see if he can duplicate the issue in their lab. All the tcpdumps in the world showing the issue only existing on the APs and not passing to the switches couldn't convince him to let me try rolling back the firmware.

Curious which access point models are you seeing this issue?

We are using MR32s two APs in a single office running around 30 people. The APs themselves are 10 Meters apart and have a fairly even split of clients connected. It's completely Meraki here, MX84 -> MS220-24P -> APs plus a number of wired devices. So it really couldn't get any simpler.
Plugging in a Cambium E500 in sorted the issue and I can clearly see that the packets over 1462 are received by the AP, but never forwarded to the switch when I did a tcpdump on the uplink port to the MX84.

I am experiencing this issue on Meraki MR 52's.  We have 35 plus deployed across 3 floors. 

MRCUR
Kind of a big deal


@RRuff1wrote:

I am experiencing this issue on Meraki MR 52's.  We have 35 plus deployed across 3 floors. 


Mention this thread/case numbers to the assigned engineer so they can get everything together if your issue is related. The more info the engineering team can gather the better. 

MRCUR | CMNO #12

I just received a response back from support. I find it hard to believe but wanted to check if it is true as my initial thoughts were "you can't be serious"

I tried reproducing the environment again and I found that L3 roaming reduces the MTU from 1500 to 1448 and I confirmed this with Engineering aswell.
So the reason why we were getting that error in your tests is because of the L3 Roaming enabled. If you change the L3 Roaming to make a part of the LAN you would be able to send pings without any issues. Let me know if there is a specific requirement for which you need it to be in L3 roaming mode.

But the whole point of L3 Roaming is to have seamless handover between APs. Or do I need to change over to bridging mode to get this working.

I've not seen any mention in any documents and the only place I have found that talks about the MTU limitation is on a Cisco forum: https://supportforums.cisco.com/t5/lan-switching-and-routing/3560x-mtus-and-icmp-unreachables/td-p/2...

The issue is that packets are silently discarded with no notification back to the wireless client and never sent to the destination.

I'm surprised this hasn't been noticed before.

MRCUR
Kind of a big deal

@PeterL L3 Roaming is only needed if your network is architected in a way that a user would be roaming among AP's which cannot bridge them to the same VLAN. I do agree that you've found a bug which shouldn't be occurring with L3 roaming enabled, but I would highly recommend moving to a more traditional deployment where you bridge clients to the LAN as I think you'll have a much better experience overall. 

 

See here for the official L3 roaming documentation that explains the use cases where it's recommended: https://documentation.meraki.com/MR/Client_Addressing_and_Bridging/Layer_3_Roaming

 

I do not believe the issue you're seeing is related to what's being discussed in this thread regarding issues with MR 25.9 firmware in Intel WiFi adapters. 

MRCUR | CMNO #12
RRuff1
Here to help

We are having this issue across all SSID's both using preshare keys, and Radius authentication.  In addition, we are experiencing extremely slow throughput causing a host of complaints across our entire enterprise.  In an attempt to fix the issue, I upgraded to 25.10 as it is a "stable release candidate".  The throughput seems to be as bad, if not worse.and the SSID issue is not as frequent but still there.

Leejohn1
Here to help

What is the recommendation for this? 🙂

 

MRCUR
Kind of a big deal


@Leejohn1 wrote:

What is the recommendation for this? 🙂

 


Roll back your firmware to the last known good version. 

MRCUR | CMNO #12

What's strange we are having this issue with clients using the pre-shared key and not radius. I would assume its related to both authenticate, here I always an error with my Asus too that easily solve by Asus Live Update you should try this.

MRCUR
Kind of a big deal

A new stable release candidate is available now as 25.11. Anyone want to give it a try that was seeing the issue with Windows machines?

 

New

  • HSRP support for Bridge Mode Layer 2 client isolation

Bug fixes

  • AP spoofs not being detected. (MR52/72)
  • A corner case where AP broadcast on W52 channel when in site survey mode (MR84)
  • A corner case where AP would stop broadcasting SSID (MR30H/33/42/53/53/74/84)
  • A corner case causes AP to reboot (MR18)
  • A corner case causes AP to stop responding to clients (MR32)
  • A corner case causes APs lose connectivity to dashboard until reboot in certain cases (MR33/74)

Known issues

  • Condition under investigation that causes VoIP RTP packet loss (MR42/MR52/MR53)
  • Condition under investigation causes 2.4GHz radios to become unresponsive (MR32/MR72)
  • Condition under investigation causes radios to become unresponsive for 5 seconds in high density networks (MR34/MR32/MR72/MR26)
  • Condition under investigation causes lower than expected throughput on the 2.4GHz radio (MR26/MR34)

Other

  • General performance and stability improvements.
MRCUR | CMNO #12

Same issue from here. We also applied the 25.11 release, but no luck. Windows clients running 7 or 10 has packet losses. The same machine when boot with linux OS does not have packet loss. Apple devices like iPhones and Mac Books are not having issues. Did someone fix this already?

 

We have a open case with Meraki, but looking further here if someone made progress.

 

Cheers!

 

Alexsandro Reimann.

This is becoming ridiculous. We also did not have problem with Mac devices (which is Linux based). 

What is there is security vulnerability and we are forced to update ? 

 

Just to ask you one more thing - do you use SonicWall for the firewall? Most of the people have problem when using SW. 

 

 

No SW here. All the network run Cisco devices. Wireless running with MR53.

MRCUR
Kind of a big deal


@AReimann wrote:

Same issue from here. We also applied the 25.11 release, but no luck. Windows clients running 7 or 10 has packet losses. The same machine when boot with linux OS does not have packet loss. Apple devices like iPhones and Mac Books are not having issues. Did someone fix this already?

 

We have a open case with Meraki, but looking further here if someone made progress.

 

Cheers!

 

Alexsandro Reimann.


Have you been able to provide Meraki with monitor mode packet captures while the issue is occurring? I know from the cases that were open no one has provided useable packet caps to help diagnose the issue. This issue is seemingly very difficult to reproduce in a lab/test environment and without monitor mode packet caps it's extremely difficult to see what's happening. 

MRCUR | CMNO #12

I work for a school district and I'm having this issue at one of my elementary schools after upgrading to 25.9.  Most of the schools have been running 25.9 with no issues, so I thought it was fine.  

 

Our teacher laptops (Dell E5550 running Windows 7 Enterprise) will randomly disconnect.  The SSIDs will sometimes not even be visible briefly.  Then they reboot their computer and everything is fine for a while.  Oddly the students running HP G3 Chromebooks in the same room won't get dropped.  

 

I figured that maybe the issue would be resolved with 25.11 coming out now, so I pushed that to this school, and the issue still occurs.

 

I provided a monitor mode capture, but of course the issue didn't come up while I was doing it!  

 

My case # is 02634264

MRCUR
Kind of a big deal

@Jmezo What kind of WiFi cards are in the Dell's? What driver version are you using with them? I know of at least one district that updated the WiFi driver on their machines and the issue went away. Unfortunately doing that at my district has not helped. 

MRCUR | CMNO #12

We have mostly Dell E7470 with Intel Dual Band Wireless-AC8260

Driver version: 19.10.10.2 - you cannot upgrade this higher on Windows 7 for some reason. 

Just FYI - similar thing occurred for Surface Books. 

 

Regards,

Filip

We are seeing the same issue with our dells e7470 and e7480

Now the really important question; what are the WiFi chipsets in the affected machines?

Do you mean WiFi card model? That would be this one

We are using the integrated wifi card that comes with Dell (I'll take the exact model, I have another session with customer to make some new tests today) and also using a USB Wifi Adapter from TP-Link. Both of them are giving packet loss when running Windows 10 or 7. With Windows 8 and linux based OS the network goes smoothless.

 

We try to deactivate client load balance feature (since it received some improvements at 25.x), but running 25.09 or 25.11, no matter the configuration applied, the scenario is the same, packet loss only at Windows 7 and 10 OSs.

 

I'll ask TAC to rollback to 24.12 if we don't get any success adjusting some settings today. Since this is a new network, it was installed initially with 25.09.

Guess what??? We made rollback to 24.12 yesterday and everything back to normal. Customer also report a better response time to ICMP tests. 

MRCUR
Kind of a big deal

@RBederian What do you have the minimum bitrate configured at for your SSID showing the issue?

 

@AReimann Quite frankly I don't think the issue you're experience is what was being discussed in this thread. 

MRCUR | CMNO #12

@MRCUR, sorry, but I did not say it before, I found logs for windows stations regarding a load balance algorithm trying to redirect the device to another AP, with fewer users associated to it, but the client got disconnect and looking for logs I found a high number of messages like '802.11 association rejected for load balancing'.

 

I did not put all the details and logs found from my side, but reading the thread, everything matched. The only part I miss is to use sonicwall firewall. 

Just wanted to see if you guys have updated the firmware and have addressed this issue?

Hi @RBederian,

 

      The original building with issues is running 24.12 since we downgrade the code. There is a new building now at customer site, running code 25.11. I'll ask the customer about performance issues and I'll reply back.

 

Cheers!

MRCUR
Kind of a big deal

We are sticking with 25.3 as the issue is still present in 25.11. The last time I spoke to the MR team they had identified what was likely causing the issue but did not have updated firmware ready. 

MRCUR | CMNO #12

Got it. Can you keep me posted when you hear back from Meraki?

MRCUR
Kind of a big deal

I'll update this thread when I hear back from the MR team. 

MRCUR | CMNO #12
MRCUR
Kind of a big deal

@RBederian Give 25.12 a try. 

MRCUR | CMNO #12

Did 25.12 address the issue?

I'm updating my clients's APs that were having issues before. We'll see how those sites do later this week, and I'll report back if I have any issues.