New MR28.6 Stable Release Candidate firmware - fixes WiFi5 wave 2 performance, MT issues and more!

[cmr]

Wireless firmware versions MR 28.6 changelog

Important note

• Meraki APs use UDP port 7351 for cloud communication and TCP ports 80 and 443 for backup communications when running MR 27 and older firmware. When running MR 28 firmware, Meraki APs will now use TCP port 443 as the primary means for cloud connectivity. In order to maintain connectivity to the Meraki cloud on MR 28+ ensure that TCP port 443 is allowed to communicate with 209.206.48.0/20 on firewalls that are deployed upstream of your Meraki APs. (Wi-Fi 6 MRs)

Legacy product notice

• When configured for this version the MR12, MR16, MR18, MR24, MR26, MR32, MR34, MR62, MR66, and MR72 will run MR 26.8.3.

New

• Additional channel support in the Indonesia regulatory domain (Wi-Fi 5 Wave 2 MRs/Wi-Fi 6 MRs)
• Additional channel support in the Israel regulatory domain (Wi-Fi 5 Wave 2 MRs)
• NBAR blocked events are logged to the Event Log (Wi-Fi 5 Wave 2 MRs/Wi-Fi 6 MRs)

Bug fixes

• MT stability and performance improvements (Wi-Fi 5 Wave 2 MRs/Wi-Fi 6 MRs)
• MT Sensors are unable to connect to gateway (MR42E, MR53E)
• Imagotag stability and performance improvements (Wi-Fi 5 Wave 2 MRs/Wi-Fi 6 MRs)
• Some Bluetooth Low Energy devices were not being detected as Bluetooth clients (Wi-Fi 6 MRs)
• 802.11k feature improvements (Wi-Fi 5 Wave 2 MRs/Wi-Fi 6 MRs)
• PMK updates now contain the MR’s management VLAN (Wi-Fi 5 Wave 2 MRs/Wi-Fi 6 MRs)
• MR could enter a state where ADDBA requests would not be initiated, resulting in reduced throughput (Wi-Fi 5 Wave 2 MRs)
• Performance improvements for MRs directly connected to an MX (Wi-Fi 6 MRs)
• Inconsistent LED behavior (MR30H, MR36H)
• General stability and performance improvements (Wi-Fi 5 Wave 2 MRs/Wi-Fi 6 MRs)

Known issues

• Sporadic packet loss & instability on Layer 3 roaming & Teleworker VPN SSID's (Wi-Fi 5 Wave 2 MRs/Wi-Fi 6 MRs)
• No DHCP response error is seen after a client performs a successful L3 roam (Wi-Fi 5 Wave 2 MRs/Wi-Fi 6 MRs)
• Meraki Auth does not work when an SSID is configured to Tunnel data to a concentrator (Wi-Fi 6 MRs)

[ww]

->: Performance improvements for MRs directly connected to an MX (Wi-Fi 6 MRs)

had bad upload speed (MR Mgig ports connected to gbit mx) but its fixed now.

[CptnCrnch]

Upgrade just went smoothly. Looking good so far!

[TomasB]

Hi, do you know when it will be published as an Stable version? We have an policy to not enroll release candidate versions...

[cmr]

@TomasB with Meraki there is no difference, it is simply that once a certain percentage of networks are using the Stable Release Candidate version, it becomes a Stable version

[TomasB]

So it would be totally safe to update the firmware to a stable release candidate version ?

[cmr]

@TomasB we are running 28.6 on our 120+ production APs with good results and have most models in service.

[Erik2]

We have found this update to be good, but we are having one issue.  One of our windows laptop models running a realtek adaptar started having issues after upgrading to the release candidate.  Normally our laptops have 15 or less connections in a 24 hour period.  After the update this number soared into the 1000s on one particular device type which we have around 100 or so devices deployed.  This included devices disconnecting and reconnecting to the same AP over and over in a matter of a couple seconds.  This lead to lagging and issues with Zoom clients especially.  Anyone else run into this?

[TomasB]

I cannot say we have han any issue. We're running with Intel WIFI6 AX201 on our Windows laptops.

From what I have seen I can see a client roaming between two AP's every min.

But, no disassociation and no disconnections.

I'm note sure why this occurs. The signal is good to both AP's.

I will try with this:

Client balancing = Disable

Enable band steering = On 

[TomasB]

I have seen lot of this on our hand held terminals running Android.

They are connected to a SSID with a static password (WPA2)

I would be happy if anyone know what can cause this issue?

The devices are in lockdown mode so there is no wrong password set by user for example:

[Carlosjr56]

Have you figure out a solution to this? Most of my laptops have Realtek cards (RTL 8822) and they are either disconnection every few seconds or roaming from different APs 

This goes on for 100's of rows for the last hour

[fcsokie1]

Yes, we are having the exact same issue with our MR46 and MR52 and Realtek adapters. No issue with our Intel cards. Currently have ticket opened with Meraki and HP. Meraki disabled 802.11k but no improvement. Seems to occur on both 2.4 and 5. AX disabled. Tried lowering roaming aggressiveness, disabled band steering and client balancing. Zero improvement so far.

[Erik2]

Yes!  We did all those things we were recommended by support with no improvement.  I will say that like usual after we did their troubleshooting they closed our case even though I kept telling them that this was still an issue.  I also did not hear anything back from our rep after emailing him multiple times and him telling me they would look into it.

[Testarossa]

Hi

Is the Wireless firmware version MR 28.6 stable for update?

Thanks

[cmr]

@Testarossa we have been using it without issues for over a month on ~125 APs across quite a few sites.

[TomasB]

I would recommend upgrading. We're running 28.6 on ~360 AP's with WPA2 PSK on one SSID and Radius on another. We have no issues with this.

[CptnCrnch]

From my point of view (and several customers too): yes

[popct]

• No DHCP response error is seen after a client performs a successful L3 roam (Wi-Fi 5 Wave 2 MRs/Wi-Fi 6 MRs)

This is a show stopper. Is the client still getting a DHCP or is this just a cosmetic error? I'm not game to roll out this firmware till this is fixed.

[fcsokie1]

Is this issue happening on only devices with Realtek wlan adapters? Because we are having the same issue but only with Realtek. Have a ticket opened with HP and it has been escalated all the up through engineering who is now working with Realtek on this.

It has caused a crippling impact on our staff and neither HP or Realtek have any workarounds to offer. Moral of the story:  avoid Realtek at all costs.

[Richard2]

Any update on this from HP/Meraki?  

We are in the same exact boat. So far we have isolated the issue to the following Realtek WLAN Cards so far: RTL8822CE and RTL8822BE. Current workaround is to install the following older drivers and this works fine for connectivity but not sure if it has other side affects. If we install the latest driver from HP, Windows Update the problem comes back.

• o RTL8822CE - 2024.0.8.134, driver date: 12/1/2021
• o RTL8822BE - 2024.0.8.133, driver date: 8/19/2021

So far we have tested the following AX disabled, 12mbps bit rate set, radio power adjustments, channel adjustments etc, max performance settings for the WLAN adapter. We have not tried disabling 802.11k yet, but does not look like that works from other users feedback but we will try it anyway. We have an active support ticket with Meraki.

if we don't use the drivers noted above, what we notice in the Meraki logs in the request "client has left ap" messages every few seconds and also, not sure if this is related is that even though we are not using 802.1x, we are seeing 802.1x authentication related events on the WiFi 6 access points. When we have the problem the packet loss is about 8% to 12%.  Intermittently as well the client would chose to connect to distant access points.

once we install the older drivers then everything seems to go back to normal, no packet loss, constantly connect to nearest access point.

Thanks.

[Erik2]

Nope. No update. They refuse to see it as an issue and have closed my case. I have even written to my rep and he said he would look into it and then never said anything back.

[fcsokie1]

Yes, I have an update. I took this issue all the way up the engineering chain at HP and after extensive wlan adapter logging, Realtek has created a new driver to address this issue. I have it in production in my environment and it is working great. They are currently working on a build to be released to the public. I don't have a timeline but help is on the way. They haven't provided me with clear explanations on the root cause other than what Meraki told me, which is that it was related to the Realtek wlan's incompatibility with 802.11k. We brought in identical devices with Qualcomm and Mediatek wlan cards and had no issues whatsoever with our Meraki APs.

[Carlosjr56]

Wow, good progress there. Let us know what the public driver version is once you find out its been released. 

[Carlosjr56]

@fcsokie1 Would you have any updates on the driver from Realtek? Is it available to the public yet? 

[katsu]

Hi

Where can I download 2024.0.8.134?
I could not find it on hp support site.

Thanks

[Richard2]

Hello,

I used the one from Lenovo's site and it works for the HPs, but you can also try from Microsoft update catalog: 

https://www.catalog.update.microsoft.com/Search.aspx?q=Realtek+8822CE+Wireless+LAN+

https://support.lenovo.com/ng/en/downloads/ds120411-realtek-wireless-driver-for-windows-10-64-bit-th...

[katsu]

Hello,
Thank you for the information.

I'll give it a try.

[Erik2]

RealTek only here

[Carlosjr56]

Same on our side. Only realtek cards are the issue. We are still running 28.5 

[Richard2]

Does anybody have the luxury to test this if the same Realtek cards have the same issue on other (non-Meraki) enterprise level wireless networks or another wireless mesh type network - or is this problem specific to Realtek and a Meraki environment?

I know on our testing if we pin a SSID to an access point (where the Realtek clients cannot roam) we don't experience any of this problems. Also if we use the same Realtek cards on our home wifi networks we don't seem to have this issue.

[Carlosjr56]

I  was told by support that the new beta MR29.1 might fix the problem.

[GIdenJoe]

Usually client drivers are the problem.

Like mediatek chips like my wife has one in her laptop have terrible drivers.

I had to check some gray websites before I finally found a firmware that fixed stability issues.

Issue was packetloss every x amount of packets for multiple seconds.  Moving the screen or laptop itself made the problem worse while moving it and improving when stationary at a certain angle.  After upgrade few times to the final version I found finally fixed this.

[Carlosjr56]

Anyone have an update on the firmware issue with Realtek? Is it no longer an issue with the new Realtek drivers? 

[Richard2]

we have not had any new findings/report of the problem since disabling 802.11k across our environment. we cannot really control installed drivers on all enduser devices and disabling 802.11k was the only system wide workaround; for those we can control the drivers then the older Realtek fw noted above is still the current workaround for us until newer drivers can be tested.

we are NOT running the beta mr29.x firmware as of yet.

we have not tested any of the later release Realtek drivers at - this may or may not contain a fix. https://www.catalog.update.microsoft.com/Search.aspx?q=Realtek+8822CE+Wireless+LAN+

hope that helps.

[Carlosjr56]

I tested Realtek 2024.0.10.226 and a newer firmware, but I had to roll it back as it was still happening. 

I have to try one of those newer one from the catalog. 

For some reason disabling 802.11k on my side, didn't seem to make a difference. 

What firmware are you running on your setup at the moment? 

[Richard2]

At this point we are running the latest stable fw 28.7.1. We also have AX disabled across the environment along with 802.11k disabled.