We just recently upgraded our Access points to the latest stable firmware. Previously we were on 24.11 and did not have any issues.
Issue we are currently seeing is affecting our Windows 7 and 10 clients. All of Mac OS X have not experienced this issue.
The client is connected to SSID using pre-shared. After the client roams user loses wireless connection. Laptop is no longer able to connect to the SSID. Also, we have seen a few cases were the laptop is no longer able to see available SSID's. After we restart the laptop the user is able to connect to SSID. Laptop will likely experience the same issue the next day.
Have anyone you guys seen this same issue with 25.9 firmware? I would like to troubleshoot this issue before I decide to roll back the firmware.
Move to the stable release candidate 25.10. It has this fix:
"APs in repeater mode erroneously broadcasted 802.11r support causing client connectivity issues during roaming.".
We are using 25.10 without any issues.
None of my AP's are in repeater mode. Also, 802.11r was disabled on this SSID and I am not mistaken Windows does not use 802.11r when using pre-shared WPA2. It would have to be configured with Radius Auth 802.11x
We're seeing this as well unfortunately. The issue is definitely only present on Windows, and perhaps more specifically with Intel wireless cards. Macs, iOS devices and Chromebooks are not exhibiting the issue.
This is happening on SSID's with 802.1x enabled with our own RADIUS servers. We're also seeing the devices unable to even see the SSID for a period of time. Updating the Intel driver has not changed anything and both Win 7 & 10 are exhibiting the issue. When this is happening, the AP logs "Client has left AP" numerous times every few seconds. Often times the event log will also show "XX events were not logged". Eventually the client can successfully connect, but the issue reoccurs eventually.
I have case 02459630 open regarding this issue but haven't really gotten anywhere yet besides moving a few AP's to a test network. We'll likely need to roll back firmware very soon as the number of devices with the issue is growing.
What's strange we are having this issue with clients using pre-shared key and not radius. I would assume its related to both authenticate.
You mentioned moving AP's to a test network. Can you elaborate what you did here?
@RBederian So far the test network is an identical config to our production template which is applied to all other MR networks. The engineer assigned to the case requested this so we can tweak settings on specific AP's to identify a root cause.
I strongly suspect at this point the issue is firmware based however. We'll see if we get any useful data from the AP's in the test network in the next day or two, but if not I'll need to roll the firmware back as this is a major issue for us.
I agree. Definitely an windows driver issue with Meraki firmware. The Radius Network SSID you're are seeing the issue do you have 2.4 and 5GHZ enabled or just 5GHz?
My case number is 02459891
Both 2.4 & 5Ghz are on but with band steering enabled.
We rolled back to firmware 24.12. Will let you know how it goes this week. Keep me posted if you guys were able to resolve the issue.
Thanks!
Have you had any issues reported again since rolling back?
I have my primary template scheduled to roll back to 25.8 (the version it was previously running before moving to 25.9) on Friday night but have kept a few AP's in a separate network where the issue is occurring. The problem at this point is that I cannot manually reproduce the issue to get a monitor mode packet cap that support wants to see.
We haven't had any reports yet with the issue rolling back to 24.12. Interesting so rolling to 25.8 you did not see this issue.
I haven't yet rolled back to 25.8, but it is the previous version that the template was set to and we were not seeing the issue until the 25.9 update.
I did notice in the 25.9 release notes that this was added (not mentioned in 25.8):
We have MR34 and MR42. We did not see the issue with the MR34. Thanks for confirming you were not able to see this issue on the 25.8 firmware. We might just wait till this issue has been resolved before updating.
Hi all. Unfortunately, we experienced the same problem on our end. Just to give you an idea from the different topic - you can track on your firewall enormous amount of the connections for the devices which eventually cause drops (firewalls have the upper limit of simultaneous connections). In our case, it is SonicWall.
Also in addition - yes, it was affecting only Windows devices, especially where we have a higher density of the APs(all our devices are MR42). We are Win7 shop so cannot say anything about Windows 10. I can also confirm this could be Intel related. We are using Latitude E7470 with Intel Dualband 8260 card. Different driver / bios versions did not make any difference. Seems like some of the phones were affected too. No problems with Mac devices.
In our case, we decided to back to previous firmware. 3 days up as of now without any big problems.
Looking forward to seeing Meraki applying fix. We might be in trouble in case there is a major security update.
I am also only seeing the issue on MR42's so far and have not had any reports from buildings with MR32's.
@Krewal & @RBederian What is the minimum bitrate set to on the SSID's you had issues with?
We have the minimum bit rate set at 18. We have also seen the issues with Windows 7. Laptops we are using are Dell 7470/7480 as well.
It mentions as a bug fix, I would assume this has not been resolved?
I'm also pretty confident MR34 are not seeing this issue.
Have one of you guys heard back from Meraki in regards to your case?
>We have the minimum bit rate set at 18.
I have seen machines refuse to connect with this setting. Make the minimum 12Mb/s for safety.
What was the firmware you guys had before updating? Would like to confirm you guys were not seeing this issue with 25.8?
Was you guys bit rate higher than 12?
@RBederianwrote:What was the firmware you guys had before updating? Would like to confirm you guys were not seeing this issue with 25.8?
Was you guys bit rate higher than 12?
We were running 25.8 prior to 25.9.
Our min. bitrate is much higher than 12. I don't think that's related to the root issue, I just asked because a support engineer mentioned it to me.
I moved all but three AP's back to 25.8 on Friday. We'll see if the reports of issues go away Monday/Tuesday as I expect they will.
Please keep me posted. Im currently on 24.12 and would to at least update to 25.8
So far I haven't had any reports of issues since rolling back to 25.8. I've moved a set of AP's that are known to show the issue to 25.10 at the suggestion of MR product management to see if the issue is corrected.
Same here. We will be putting an AP on 25.10 as well and test. Will follow up if we noticed the same issue
How did you get access to 24.12? That is what I was previously using and only upgraded because the AP's were randomly rebooting. I would gladly go back to that issue at this point
you would need to reach out to Meraki support to downgrade firmware
Just talk to Meraki to roll back the firmware to previous stable for you.
Depending on how long ago you did this you can roll back the firmware upgrade yourself. Go:
Organization/Firmware Upgrades
Click on the blue circle to roll back the change.
Meraki mentioned this issue might of been addressed with the 25.10. I currently have one access point with this firmware and has not seen this issue. Have anyone of you guys updated to 25.10?
I haven't had any issues with 25.9 or 25.10. However I updated a bunch of access points to 25.10 a couple of weeks ago after reading about other people having problems. I'm still not having any issues.
I'm moving everything to 25.10 today after a successful test on four AP's.
Please keep me posted on how 25.10 goes
I am having a semi-similar issue which may indicate that I need to do the firmware upgrade to 25.10 as well.
Started at a new company where we are having an MTU issue where I can't have a packet size greater than 1462 over wireless. The easiest way to replicate this is using ping. If I use windows with a "ping 8.8.8.8 -l 1420 -f" (as windows take the 42 bytes for the ICMP packet) and I get a response, but 1421 and I don't.
After various tests I have isolated the issue to the wireless network as I don't get this issue over the Switch -> Router -> Internet and don't have this issue on other wireless networks so have ruled out my laptop.
Since the firmware was upgraded over 2 weeks ago I can't roll back but there has been various complaints of certain sites being unavailable and VPN connections not working "for the last month or so" which to me sounds rather coincidental.
Planning to take a non-Meraki AP onsite tomorrow and have that option to test, plus upgrade to 25.10 when I have a failback position with a secondary AP as the site principally uses wireless for all devices and there is only two APs.
I updated to 25.10 and cannot speak to the SSID issue, as I rolled back within a couple days because throughput was horrible. I am pending a rollback to 24.12 right now.
@RBederian I'm rolling everything back to 25.8 tonight. Issue has not been fixed on 25.10.
Has anyone else seeing this issue with 25.10?
I'm continuing to see this issue in 25.8 although much less often than 25.9/25.10.
Is there a way to roll back to 25.8 if it has been over 2 weeks as the Meraki portal doesn't allow rollback if it has been longer than 2 weeks?
Is there a way to force the downgrade as I am not familiar with the Meraki product suite and it isn't clear there is a way to do it?
You would need to reach out to Meraki support
Open a case with Meraki and request a rollback
Curious which access point models are you seeing this issue?
I am experiencing this issue on Meraki MR 52's. We have 35 plus deployed across 3 floors.
@RRuff1wrote:I am experiencing this issue on Meraki MR 52's. We have 35 plus deployed across 3 floors.
Mention this thread/case numbers to the assigned engineer so they can get everything together if your issue is related. The more info the engineering team can gather the better.
I just received a response back from support. I find it hard to believe but wanted to check if it is true as my initial thoughts were "you can't be serious"
I tried reproducing the environment again and I found that L3 roaming reduces the MTU from 1500 to 1448 and I confirmed this with Engineering aswell.
So the reason why we were getting that error in your tests is because of the L3 Roaming enabled. If you change the L3 Roaming to make a part of the LAN you would be able to send pings without any issues. Let me know if there is a specific requirement for which you need it to be in L3 roaming mode.
But the whole point of L3 Roaming is to have seamless handover between APs. Or do I need to change over to bridging mode to get this working.
I've not seen any mention in any documents and the only place I have found that talks about the MTU limitation is on a Cisco forum: https://supportforums.cisco.com/t5/lan-switching-and-routing/3560x-mtus-and-icmp-unreachables/td-p/2...
The issue is that packets are silently discarded with no notification back to the wireless client and never sent to the destination.
I'm surprised this hasn't been noticed before.
@PeterL L3 Roaming is only needed if your network is architected in a way that a user would be roaming among AP's which cannot bridge them to the same VLAN. I do agree that you've found a bug which shouldn't be occurring with L3 roaming enabled, but I would highly recommend moving to a more traditional deployment where you bridge clients to the LAN as I think you'll have a much better experience overall.
See here for the official L3 roaming documentation that explains the use cases where it's recommended: https://documentation.meraki.com/MR/Client_Addressing_and_Bridging/Layer_3_Roaming
I do not believe the issue you're seeing is related to what's being discussed in this thread regarding issues with MR 25.9 firmware in Intel WiFi adapters.
We are having this issue across all SSID's both using preshare keys, and Radius authentication. In addition, we are experiencing extremely slow throughput causing a host of complaints across our entire enterprise. In an attempt to fix the issue, I upgraded to 25.10 as it is a "stable release candidate". The throughput seems to be as bad, if not worse.and the SSID issue is not as frequent but still there.
What is the recommendation for this? 🙂
@Leejohn1 wrote:What is the recommendation for this? 🙂
Roll back your firmware to the last known good version.
What's strange we are having this issue with clients using the pre-shared key and not radius. I would assume its related to both authenticate, here I always an error with my Asus too that easily solve by Asus Live Update you should try this.
A new stable release candidate is available now as 25.11. Anyone want to give it a try that was seeing the issue with Windows machines?
Same issue from here. We also applied the 25.11 release, but no luck. Windows clients running 7 or 10 has packet losses. The same machine when boot with linux OS does not have packet loss. Apple devices like iPhones and Mac Books are not having issues. Did someone fix this already?
We have a open case with Meraki, but looking further here if someone made progress.
Cheers!
Alexsandro Reimann.
This is becoming ridiculous. We also did not have problem with Mac devices (which is Linux based).
What is there is security vulnerability and we are forced to update ?
Just to ask you one more thing - do you use SonicWall for the firewall? Most of the people have problem when using SW.
No SW here. All the network run Cisco devices. Wireless running with MR53.
@AReimann wrote:Same issue from here. We also applied the 25.11 release, but no luck. Windows clients running 7 or 10 has packet losses. The same machine when boot with linux OS does not have packet loss. Apple devices like iPhones and Mac Books are not having issues. Did someone fix this already?
We have a open case with Meraki, but looking further here if someone made progress.
Cheers!
Alexsandro Reimann.
Have you been able to provide Meraki with monitor mode packet captures while the issue is occurring? I know from the cases that were open no one has provided useable packet caps to help diagnose the issue. This issue is seemingly very difficult to reproduce in a lab/test environment and without monitor mode packet caps it's extremely difficult to see what's happening.
I work for a school district and I'm having this issue at one of my elementary schools after upgrading to 25.9. Most of the schools have been running 25.9 with no issues, so I thought it was fine.
Our teacher laptops (Dell E5550 running Windows 7 Enterprise) will randomly disconnect. The SSIDs will sometimes not even be visible briefly. Then they reboot their computer and everything is fine for a while. Oddly the students running HP G3 Chromebooks in the same room won't get dropped.
I figured that maybe the issue would be resolved with 25.11 coming out now, so I pushed that to this school, and the issue still occurs.
I provided a monitor mode capture, but of course the issue didn't come up while I was doing it!
My case # is 02634264
@Jmezo What kind of WiFi cards are in the Dell's? What driver version are you using with them? I know of at least one district that updated the WiFi driver on their machines and the issue went away. Unfortunately doing that at my district has not helped.
We have mostly Dell E7470 with Intel Dual Band Wireless-AC8260
Driver version: 19.10.10.2 - you cannot upgrade this higher on Windows 7 for some reason.
Just FYI - similar thing occurred for Surface Books.
Regards,
Filip
We are seeing the same issue with our dells e7470 and e7480
Now the really important question; what are the WiFi chipsets in the affected machines?
We are using the integrated wifi card that comes with Dell (I'll take the exact model, I have another session with customer to make some new tests today) and also using a USB Wifi Adapter from TP-Link. Both of them are giving packet loss when running Windows 10 or 7. With Windows 8 and linux based OS the network goes smoothless.
We try to deactivate client load balance feature (since it received some improvements at 25.x), but running 25.09 or 25.11, no matter the configuration applied, the scenario is the same, packet loss only at Windows 7 and 10 OSs.
I'll ask TAC to rollback to 24.12 if we don't get any success adjusting some settings today. Since this is a new network, it was installed initially with 25.09.
Guess what??? We made rollback to 24.12 yesterday and everything back to normal. Customer also report a better response time to ICMP tests.
@RBederian What do you have the minimum bitrate configured at for your SSID showing the issue?
@AReimann Quite frankly I don't think the issue you're experience is what was being discussed in this thread.
@MRCUR, sorry, but I did not say it before, I found logs for windows stations regarding a load balance algorithm trying to redirect the device to another AP, with fewer users associated to it, but the client got disconnect and looking for logs I found a high number of messages like '802.11 association rejected for load balancing'.
I did not put all the details and logs found from my side, but reading the thread, everything matched. The only part I miss is to use sonicwall firewall.
Just wanted to see if you guys have updated the firmware and have addressed this issue?
Hi @RBederian,
The original building with issues is running 24.12 since we downgrade the code. There is a new building now at customer site, running code 25.11. I'll ask the customer about performance issues and I'll reply back.
Cheers!
We are sticking with 25.3 as the issue is still present in 25.11. The last time I spoke to the MR team they had identified what was likely causing the issue but did not have updated firmware ready.
Got it. Can you keep me posted when you hear back from Meraki?
I'll update this thread when I hear back from the MR team.
Did 25.12 address the issue?
I'm updating my clients's APs that were having issues before. We'll see how those sites do later this week, and I'll report back if I have any issues.
@RBederian It should, yes. Give it a test and if you still have issues, ask Support to disable CCX completely.
How's the new firmware doing? Has it resolved this issue? I was thinking about upgrading it tomorrow and wanted to see how it has been for you guys so far
I have 25.12 running on ~1,400 AP's and so far have not had any complaints of random disconnects. This is only the second day with teachers back in the district however. Hopefully it's really fixed.
Hi,
Just picked up on this thread as we are experiencing similar issues in our environment. It seems to be in a single location but looks like client is trying to switch APs and that causes the issue.
Seeing the following in the event log on an impacted machine (Dell).
The network interface "Intel(R) Dual Band Wireless-AC 8260" has begun resetting. There will be a momentary disruption in network connectivity while the hardware resets.
Reason: The network driver requested that it be reset.
This network interface has reset 13 time(s) since it was last initialized.
Is this message found for others with this issue? Our SSID is 802.1X based with machine/user certificates for authentication.
Users reporting random disconnections and short periods of dropped access.
Currently running 25.11 and considering trying 25.12 before opening a support request to go pre-25.9.
Keep in mind you can only revert firmware for 1 week, if I remember correctly.
This is becoming ridiculous by now. half of the year yet they could not address it?
@Krewal wrote:Keep in mind you can only revert firmware for 1 week, if I remember correctly.
This is becoming ridiculous by now. half of the year yet they could not address it?
Support can modify firmware version on the network at any time. The issue being discussed in this thread is believed to be resolved in 25.12.
@GaryM wrote:The network interface "Intel(R) Dual Band Wireless-AC 8260" has begun resetting. There will be a momentary disruption in network connectivity while the hardware resets.
Reason: The network driver requested that it be reset.
You should update the driver directly from Intel. What you're seeing is very likely a driver issue and unrelated to what is being discussed in this thread.
@GaryM You would see the AP log clients leaving because your clients are disconnecting from (leaving) the AP. But since you are seeing the client side WiFi adapter reset at the request of the driver, it seems like a client side driver issue. We never saw the client driver request a reset throughout this issue.
It wouldn't hurt to move to 25.12 firmware since it is the stable release candidate and will likely become the stable release shortly.
Haven't updated my status on this in a while. At my original problem location I downgraded to 24.12 and everything has been fine since.
Now in May 2018, another network was upgraded to 25.9 and the problem started occurring there. I didn't trigger the upgrade, so I'm not sure what did.
Anyways, I tried upgrading the firmware to 25.11 and the problem persisted. Then on Friday 8/24 I tried to update the firmware to 25.12. Now this morning I'm getting even more reports of issues with disconnecting.
The problem is with our staff laptops. They're Dell e5550 with Intel Dual Band Wireless-AC 7265 adapters. I know the machines aren't up to date on the latest Intel drivers, but mine is. I went to that building this morning and was disconnected 3 times in an hour.
At this point I've got the downgrade to 24.12 scheduled to run tonight.
@Jmezo Since you can reproduce this pretty consistently, I'd recommend contacting Support and asking them to disable CCX completely on the SSID you're having issues with while still on 25.12 firmware. The thought is CCX has been causing these issues. Part of it was disabled in 25.12, but perhaps not enough to fix this completely.
If you do this and have a case number, feel free to message it to me and I will make sure it gets in front of the product team.
I know this is an old thread, but our issues appear to have been resolved by moving to 25.12
We have not made any request for CCX extension disabled as yet. We have not made any driver updates beyond one machine.
We had a significant number of complaints relating to a single area on one of our buildings. Logs seems to indicate clients moving between APs constantly at times.
We have been on 25.12 just over 1 week.
This might all be irrelevant to thread, but I thought I would share experience in case anyone sees similar. All APs are MR42.
Unfortunately the CCX changes in 25.12 did not resolve the issue for us. We also tried completely disabling CCX/Fastlane which did not resolve the issues either. We've now modified the min. bitrate for our primary SSID to see if that helps (no complaints so far).
I put the minimum Bitrate to 12MBit. Runningversion 25.12. No complaints the last days.
@MRCUR Do you have any MR34 running on 25.12? I need to find out if they fixed the DHCP issue with a WPA2 passcode SSID without being a guinea pig again.
@Asavoy We have lots of MR34's on 25.12 but the only SSID with WPA2 uses Meraki DHCP (which is working).
My sample size is much smaller, but it was limited to sites with very vocal users and one or two APs at each. Been running 25.12 without complaint.
I see the same problem with windows clients. Now i am on 25.12 on MR18 but still the same issue, so now i am waiting for support to disable ccx extensions to see, if this makes any difference.
Having a similar issue on an SSID with WPA2 preshared key on Dell laptops with Intel cards. Seems making my channel width 20Mhz worked previously so going to try again with this particular network. Currently running 25.13.
I haven't had the issue after 25.12 when I updated. I'm running similar hardware on a couple systems, Dells with Intel cards. Though, we have way more HP these days, so it's not a good measure to know for sure.
One thing to remember, especially with Dell laptops, is that the OEM uses some proprietary software/drivers that don't always get along with WAPs. If a Dell has the "Unified Wireless Driver" software (or whatever it's called), you should definitely uninstall that POS software, uninstall the driver, reboot and let Windows reinstall the drivers. Unfortunately, most of the time you won't be able to update the driver and are stuck with whatever 3 year old driver Dell has released. I refuse to use HP products, so not sure if they're still as bad as Dell for that.