Meraki MR46 and Lenovo E480 (with Realtek 8822BE Wireless LAN 802.11ac) packet loss problem

[MedarV]

Dear All, 

I have a significant problem communicating with MR46 and Realtek wifi adaptor. During communication always appears request time out while pinging. 

We have 220 Lenovo E480 (%80 of all laptops) and 101 AP with Meraki MR46. Realtek adaptor driver which is last version requested on Lenovo web site (version 2024.0.8.122) is not working properly. It causes time out. 

We are using meraki system in school. Zoom is our most using program during this period. If packet loss appears then Zoom starts to freeze and people cannot finish their meeting with good quality. 

We found another older driver for realtek and I guess this driver produced for Asus laptops. Driver version is 2023.72.1215.2017. We are using this driver for two days. It seems working better but we are not sure that we will face another problem or not. 

And we found another solution to change realtek wifi adaptor with Intel (which is using in Lenovo E490) then E480 is working with Meraki without problem. But it is expensive and time losing opration to change 220 laptop's wifi adaptor. 

Cisco dealing with this problem but they still haven't addressed the problem correctly. We started to deliver old driver for realtek for all E480 laptops. Still we are not sure. 

I want to ask that is there any other permanent solution on meraki site (e.g. changing some adjustment in meraki side)

Thanks in advance. 

[cmr]

@MedarV what Realtek chipset is it?  On the realtek website the 2023 version seems the latest for most NICs and is from 2017. 

As the NICs are WiFi5 standard you can disable the WiFi6 features on the APs from the dashboard, that is pretty much the only thing you can do from the Meraki side for this issue.  It is a simple change so might be worth it even if it is done temporarily while making other changes to the devices.

From the client side with the 2024 driver try disabling all power saving options as that worked for older Intel drivers on their WiFi5 NICs, there was a bug in the way the older software worked.

[MedarV]

@cmr , First of all thank you very much for your quick answer. 

Our Realtek chipset model is RTL8822BE. 

While we were doing test with Cisco Meraki engineers they offer us to disable wifi 6 and they created another SSID just for test and we did same test that you offered. First it seems it is working normally. But several hours later user informed that she has still same problem. 

Secondly, We also disabled power safe mode on some test laptops from realtek device manager settings. But it isn't solve the problem. 

Thanks. 

[redsector]

I am really happy now with firmware 29.3 on all my networks with an amount of different MR accesspoints.

[redsector]

We had issues too, but with Intel chipsets. But since firmware 28.5 is installed on the MR accesspoints we have no problems any more.

[MedarV]

ok. Thank you. We will try with several AP.. I will share the result

[JimmyM]

Hi MedarV

We experienced the exact same issue 3 years ago when we sold a batch of Thinkpad E480 and E580.

I had to use one myself to see what my clients were going through.

Meraki, Fortinet, Unifi, personal home AP.... The experience with the Realtek chip on this laptop was horrible.

At the time there were several topics on some forum on this subject.

After weeks of troubleshooting and pain, we decided to change all customer Wifi Chip on E480 and E580 for Inter 9260 

https://www.amazon.ca/dp/B07NY2SQ5M/ref=cm_sw_em_r_mt_dp_VQ5B5Z710GCN0R4AC2ES?_encoding=UTF8&psc=1

Problem solved.

See this video for instruction to open the laptop. https://youtu.be/IDkoD0iF-Xs?t=222  

Now i have a ThinkPad E15 Gen 3 with a RTL8852AE 802.11AX. And it work perfectly.

Regards,

[MedarV]

Thank You JimmyM. We have also addressed this problem changing E490 chip (Intel) with E480 chip (realtek) After this changing we haven't faced any problem neither E490 nor E480. It is very interesting. Because  realtek worked better with E490. 

We have also several E580 and we have also problem with E580 as you told. 

We finished driver changing for realtek with older version which is 2023.72.1215.2017. Our E480's have been working with this driver for one week. It seems that this version works better than latest driver in Lenovo support page during Zoom meetings. 

Tomorrow we will update MR46's with Beta version 48.5 as Redsector suggested before. We read many article with 28.5. People are saying It is solving such kind of problems even it is Beta version.

Regards,   

[cmr]

@MedarV with Meraki a beta is often just an early release of a stable firmware.  Take the MX 16.14 release that moved from beta to stable release candidate and may well move to stable.

We run 28.5 on over 120 APs around our venues and have found it more stable than the 27.x releases.

[MedarV]

As I said before we found another solution about this problem. We used older driver for realtek and I guess this driver produced for Asus laptops in 2018 . Driver version is 2023.72.1215.2017. We are using this driver for two weeks. It seems that the problem has been solved with this driver. And we made some changes on windows registry to stop automatic update for this driver. 
  
On the other side we upgraded the MR 46 AP's with firmware 28.5. We are even testing this firmware with official current Lenovo driver for realtek version 2024.0.8.122. Meraki 28.5 firmware seems better. But it is early to say it is solved completely.  

[MedarV]

Today We faced new problem with the Realtek driver 2023.72.1215.2017. Wi-fi adaptor suddenly stops the searching SSID and Laptops cannot see any SSID anywhere. we are disabling and enabling the network adaptor with admin account. I am pushing Meraki side to define this problem. Because it seems that it will not solve like this kind of temporary solution. If any of you knows facing problem with Lenovo Realtek wi-fi adaptor and MR46 then please write your problem here. I am communication with Meraki Helpdesk with the problem. 

[ccoho]

Its not just lenovo, its all 8822BE/CE drivers.  We use HP and are experiencing the same with Meraki APs (not just MR46, multiple models).  Driver rollback worked (2024.0.10.129 ver. on the 8822CE), but as you said we also are experiencing the WLAN autoconfig service crashing using the driver you mentioned.  I'm going to try 2024.0.8.101 next to see if its old enough to eliminate the meraki bug, but new enough that it isn't crashing the WLAN autoconfig.

[TheCongQTM]

I have the same problem with 8822CE, I still can't fix it

[ccoho]

Rolling back to driver 2024.0.10.129 ver. on the 8822CE chipset has worked for me with no issues (still searching for an 8822BE driver that works and doesn't cause other issues).

[MedarV · Accepted solution]

Last week Lenovo released new driver update for E480 and E580. We are starting to test this driver for several laptops. I will write my comments after our test. If you have same problem you can also try this update. Cisco also recommend us to try this version answering our ticket about this problem. 

Please share your experience with new driver 2024.0.8.133

https://pcsupport.lenovo.com/us/en/products/laptops-and-netbooks/thinkpad-edge-laptops/thinkpad-e480... 

Regards,  

[MedarV]

It seems, this driver has been solved the problem. 

[AABREW001]

Hi, all.

Reading this thread, I wish Realtek had solved this issue. I'm running a bunch of HP 250 G4's with the Realtek RTL8821CE cards and f/w load 2024.0.10.217. Connection drops after a few minutes. Tried regressing to earlier versions of their f/w but no go. My AP's are on MR 28.5. There is no single obvious cause. Windows diagnostics will blame everything it can, like can't find gateway, or there is a problem with the card (but does not always specify what the problem is). We a school and BYOD. The kids that have HP's are also dealing with this. Is this a protocol issue or something else. Getting desperate for a viable (not costly) fix.

Thanks, in advance.

[MedarV]

Good day, 

Do you have a chance to try Realtek driver for Lenovo. Maybe it can work with HP 250 either. 

https://pcsupport.lenovo.com/us/en/products/laptops-and-netbooks/thinkpad-edge-laptops/thinkpad-e480...

Regards, 

[AABREW001]

Hello.

I will have a look at that. I also have an open ticket that is exploring other options. A reboot of AP's in affected areas also relieves the issue. Not sure how long this action will hold. There is also some discussion about 802.11k. We shall see.

[Richard2]

Any update on this from HP/Meraki?  

We are in the same exact boat. So far we have isolated the issue to the following Realtek WLAN Cards so far: RTL8822CE and RTL8822BE. Current workaround is to install the following older drivers and this works fine for connectivity but not sure if it has other side affects. If we install the latest driver from HP, Windows Update the problem comes back.

• o RTL8822CE - 2024.0.8.134, driver date: 12/1/2021
• o RTL8822BE - 2024.0.8.133, driver date: 8/19/2021

So far we have tested the following AX disabled, 12mbps bit rate set, radio power adjustments, channel adjustments etc, max performance settings for the WLAN adapter. We have not tried disabling 802.11k yet, but does not look like that works from other users feedback but we will try it anyway. We have an active support ticket with Meraki.

if we don't use the drivers noted above, what we notice in the Meraki logs in the request "client has left ap" messages every few seconds and also, not sure if this is related is that even though we are not using 802.1x, we are seeing 802.1x authentication related events on the WiFi 6 access points. When we have the problem the packet loss is about 8% to 12%.  Intermittently as well the client would chose to connect to distant access points.

once we install the older drivers then everything seems to go back to normal, no packet loss, constantly connect to nearest access point.

Thanks.

[Richard2]

Any update on this from HP/Meraki?  

We are in the same exact boat. So far we have isolated the issue to the following Realtek WLAN Cards so far: RTL8822CE and RTL8822BE. Current workaround is to install the following older drivers and this works fine for connectivity but not sure if it has other side affects. If we install the latest driver from HP, Windows Update the problem comes back.

• o RTL8822CE - 2024.0.8.134, driver date: 12/1/2021
• o RTL8822BE - 2024.0.8.133, driver date: 8/19/2021

So far we have tested the following AX disabled, 12mbps bit rate set, radio power adjustments, channel adjustments etc, max performance settings for the WLAN adapter. We have not tried disabling 802.11k yet, but does not look like that works from other users feedback but we will try it anyway. We have an active support ticket with Meraki.

if we don't use the drivers noted above, what we notice in the Meraki logs in the request "client has left ap" messages every few seconds and also, not sure if this is related is that even though we are not using 802.1x, we are seeing 802.1x authentication related events on the WiFi 6 access points. When we have the problem the packet loss is about 8% to 12%.  Intermittently as well the client would chose to connect to distant access points.

once we install the older drivers then everything seems to go back to normal, no packet loss, constantly connect to nearest access point.

Thanks.

[Richard2]

Does anybody have the luxury to test this if the same Realtek cards have the same issue on other (non-Meraki) enterprise level wireless networks or another wireless mesh type network - or is this problem specific to Realtek and a Meraki environment?

I know on our testing if we pin a SSID to an access point (where the Realtek clients cannot roam) we don't experience any of this problems. Also if we use the same Realtek cards on our home wifi networks we don't seem to have this issue.

[JimmyM]

Same thing with Fortigate AP 221.

[AABREW001]

It seems this continues to be a problem. I have no opportunity to test this on another Enterprise level network. Works at home just fine. Works on my Ubiquiti WiFi6 network no problem. Is it just Meraki? No idea. I have offered to allow some testing on my network, but have not heard back. The last snippet of information was something to do with radio management and roaming from one AP to another.

[Richard2]

Thanks - this is still problem for us as well. At this point the only work-around is using the older Realtek drivers.

Did you end up trying disabling 802.11k? If so what was the outcome?  we have a pending support request with Meraki and this is one of the items we have not tested yet.

[AABREW001]

It is disabled. Has it cured the issue? Not entirely. I've seen a drop in support requests, tho. It might be a partial fix but I would like Meraki to analyze this issue more and I'm willing to be a Guinea pig -- to a point.

[AABREW001]

Before I forget (again), there is one more thing I've tried. On the afflicted PC and in range of an AP that won't allow connection (secured, no internet), go into the wifi settings and "forget" the wifi network SSID that's misbehaving. Manually reassign it and you will connect no problem. It really smells of some form of corruption associated with the Realtek radio. IMHO, YMMV, Best of luck.

[Richard2]

Thanks - we have tried that as well and it seems to temporarily resolve the issue, the issue does come back eventually. Rebooting the client also sometimes make it behave better but often than not the problem does resurface.  

In the meantime - if they are managed devices then we can install the older Realtek drivers that seems to work reliably. If they are not managed devices then.

Just curious - are you having the same problems on non WiFi 6 Meraki access points and the Realteks? In our environment it seems the problems are tied to WiFi 6 capable access points (MR45 and MR44) - with AX enabled or AX disabled. 

[AABREW001]

All of our AP's are Wifi6 MR46 with f/w MR 28.6

A brand new network, AX disabled. We are a BYOD school and we have students with some really "vintage" computers running about the place. All sorts of connection issues with AX turned on. (Unfortunately.) Haven't tried turning it back on lately. Too many other irons in the fire.

[Richard2]

Disabling 802.11k on our testing seems to have made a difference. The Realtek adapters with any version of driver installed is now behaving better - consistently connecting to the closest AP, no packet loss, "client left ap" on the event logs are gone as well.

Disabling another feature is far from ideal but I guess its better than rolling back drivers. 

We are waiting for further information from support.

[AABREW001]

Very good. Nice to know that at least part of the problem can be reproduced.

I supposed it will be a conversation that Realtek and Meraki need to delve into.

Can't wait to see what they come up with.

Thanks for sticking with this.

[ccoho]

Where is the option to disable 802.11k in the dashboard? I'd be interested in testing this as well

[Richard2]

802.11k setting is not accessible in the dashboard - you have to call support and have them disable it on the backend. Thanks.

[redsector]

No issues with MR32, MR33, MR34, MR36, MR42, MR42E, MR44, MR45, MR46, MR46E, MR52, MR55, MR84.

Different networks with different firmware levels (most ver MR28.6)

Running Dell laptops (various types and ages), iPhones, iPads, Mac-books ...

(only thing: fading blue LED light at MR32, MR33, MR34)

[Macguy]

LOL the awful fading blue LED light... All our MR32's are so bad you can't even tell they are on.  All our MR33 lights are still bright. 

[Richard2]

There’s an update on this issue on a seperate thread in case you are not aware - most promising updates so far.

https://community.meraki.com/t5/Wireless-LAN/New-MR28-6-Stable-Release-Candidate-firmware-fixes-WiFi...

[Richard2]

this is the update im referring to

Update by fcsokie1

"Yes, I have an update. I took this issue all the way up the engineering chain at HP and after extensive wlan adapter logging, Realtek has created a new driver to address this issue. I have it in production in my environment and it is working great. They are currently working on a build to be released to the public. I don't have a timeline but help is on the way. They haven't provided me with clear explanations on the root cause other than what Meraki told me, which is that it was related to the Realtek wlan's incompatibility with 802.11k. We brought in identical devices with Qualcomm and Mediatek wlan cards and had no issues whatsoever with our Meraki APs."

[AABREW001]

OH! This is great news!!

Thank you, thank you.

Can't wait for the public release to see what happens here.

[ccoho]

Any update on a new RealTek driver to resolve this issue?

[AABREW001]

Well, assuming it has been released, I can't say I've seen it. In a BYOD environment it's tough to tell. My HPs assigned to faculty have not shown any progress through the HP Support Assist. having stated that, the number of incidents have been reduced.

[ccoho]

I've not seen a new version via Windows update or HP update (or HP's website).  Realtek doesn't seem to even have a download page for that chipset.

[redsector]

Meraki firmware update.

[ccoho]

For us at least, so far none of the stable MR firmwares have solved this issue.

[redsector]

helped me a lot solving some issues: "General stability and performance improvements"

Wireless firmware versions MR 29.2 changelog

Bug fixes

• General stability and performance improvements
• AP performance may degrade over time (Wi-Fi 6 APs)
• AP may stop broadcasting beacons on WPA2 configured SSID (MR53E)
• AP may lose Cloud connectivity for an interval of time (Wi-Fi 6 APs)
• SES-imagotag ESL status may display as gray even when ESL is functioning normally (Wi-Fi 5 Wave 2 and Wi-Fi 6 APs)
• SES-imagotag 2.4GHz Wi-Fi Co-Existence might not work on MR36H

Wireless firmware versions MR 29.3 changelog

Bug fixes

• General stability and performance improvements
• Intermittent low power mode (MR53, MR56)
• SES-imagotag application bootloop (MR30H, MR33)
• Honeywell printers unable to connect to hidden SSID (Wi-Fi 6 APs)

Wireless firmware versions MR 29.4 changelog

Bug fixes

• General stability and performance improvements
• ‘No DHCP response’ error using L3 Roaming
• ‘DNS misconfigured’ error while using Umbrella DNS

[ccoho]

I haven't installed the beta firmwares.  It is possible this helps, but we are also experiencing the issue with security appliances with built-in wireless.  Unsure if the beta MX firmwares include the same updates.