We have replaced our MR32 for a mix of MR42/MR52 and since this change, we can see that performance (bandwidth) slowly getting lower and lower and almost tend to less than 1 Mbps. Only a reboot of the device permit to have good performance back for some time.
We are observing this behavior in environment where there is multiple other SSIDs broadcasted by other third parties in the buildings next to us.
We are in 5GHz only, 40 Mhz width channel. The 5 Ghz Channels used are not overloaded in utilization by other third parties.
If i put a MR32 at the same place, same channels, i don't have the issue.
It's like MR42/MR52 have an issue when the environment is used a lot even if it is not on the channels we are using. And, a reboot solve the issue for some hours (and we keep the same channels, so it's not a question of channels overloaded).
We have open ticket to support but upgrading to last firmware does not help.
Thanks in advance.
Try setting the minimum connect speed to 12Mb/s (unless you are using something higher already). Perhaps legacy protocols are consuming a lot of airtime, or devices are holding onto APs some distance away.
Also note it takes 24 hours before the final channel plan and power levels gets put into affect. The APs slowly drift towards their final values.
If you do a speedtest between the AP and the user (using my.meraki.com) is the result also poor?
Are you using the 26.x firmware?
Thanks for your answer.
I am already with minimum connect speed 12 Mb/s.
When the problem occurs, the speedtest using my.meraki.com show a result between 0 and 1 Mbps. So users can't do anything in Wifi.
We are using the 26.6.1 firmware, we had the same issue in 25.13.
This behavior is seen only on MR42/MR52 access point. I don't have this issue with MR32 access point. Seems something related to MR42/MR52 model
My only way to solve it temporarily is to reboot the access point.
Also enable DFS channels unless you are near a radar user. Without DFS in Europe there are only two 5GHz channels, so the interference can slow them right down.
Thanks for your answers guys !
Regarding the channel utilization, it is always pretty fine on the channels used by the access points (40 Mhz), so it's seems not to be related to a high utilization of the channels currently in use by the access point itself.And if i replace the MR42 or MR52 with a MR32 with the same channels, i don't have the issue.
So it's seems related only to these 802.11ac wave 2 models and not especially if the channels used are highly used but also if all the channels around are high used (even if not overlapping with the channels currently in use).
I was not able to test many things since lockdown but i have seen that in the release notes of the 26.8 firmware, an issue fixed can correspond to my problems:
-> MR would get into a state where it would cease being able to TX frames reliably in environments with sustained high channel utilization (802.11ac Wave 2 MRs)
This corresponds to the fact that my issues are only seen on these MR42 and MR52 (802.11ac wave 2) and not on old MR32.
I will upgrade and check how it goes.
Did you ever make any headway with this issue?
We have a lot of MR42's and are now also seeing very poor performance with them until they are rebooted.
We have checked out the switching environment, check our WAN usage, and have gone over our RF profiles and cannot see the cause of the issues.
Like yourself, rebooting the AP returns it back to good performance, but with 180 across our site, doing this regularly with what is supposed to be enterprise hardware is not ideal!
We are running the latest firmware as well.
Hi @TASIS_IT , when you say the latest, can you be more specific, there is 27.6 (which I guess you mean), but others out there as well?
Yes, we are running 27.6, across all of our MR42's
Nothing else is currently available on the Merkai Portal.
After weeks of sending screenshots and packet captures, I received this response from Meraki.
I have managed to find the issue that is causing the slow speed problem over time and after a reboot, the issue just goes.
Thanks to the packet capture taken from the working and AP and broken AP we identified that the issue was being caused because the client was sending aggregated frames and the AP wasn't.
Initially, both the AP and the client decide to send aggregate frames then all of a sudden after a while the AP doesn't send aggregate frames. Hence when you restart the AP it's all fresh and negotiates with the client again to send aggregate frames and the problem disappears.
We are currently working, on fixing this issue on a feature firmware release with our internal team, however, at this point, the problem is still premature and we are looking more into why the AP goes into stops sending aggregate frames. We know that the issue came around firmware version 27.x. Our internal team has asked me to ask you if you would be willing to test our beta MR 28.1 to see if the problem also persists on this firmware.
If you are willing to upgrade a couple of APs to MR28.1 to test let me know and I can arrange a session to ping the firmware on the APs you want to test on without having to upgrade the whole network to MR28.1
I am just waiting for our summer holidays to start and then I will be in a position to test MR28.1 on a few of the AP's.
I was pleased to see that they had identified a possible source of the issue since we have been chasing this for weeks and questioning ourselves whether it was something that we had done or something we were missing with the configuration of our network.
Hopefully, the issue is resolved in the next firmware update, but ill update you all again once i have tested further.
We are running 28.1 on most of our 100+ APs and at least 30 of those are MR42/52 models.
It has been stable so far, I'll let you know if we see any problems.
How long have you been on 28.2? We moved to 28.1 and then 28.2
However I have just had exactly the same issue out of firmware version 28.1 . Slow speed between AP and client verifying with my.meraki.com and doing the speed test.
It was upgraded June 22nd, and today well its July 15th.
For an expensive AP it lasts less time than some cheap AP.
Also I have noticed on the latest MR56 after a few months its dumping clients off trying to connect it. There seems to be no light at the end of the tunnel at all with these APs.
Meraki, you really need to sort this out.
Has the beta firmware improved anything? I just noticed there is a 28.4 beta release and I am thinking of trying that.
We have a total of 22 MR42s and have been dealing with the exact issue for almost a year now. Super slow network and a lot of 802.11 disassociations with "unknown error", where clients are disconnected from the network.
If that is the case, that sort of pokes a hole in the 'this issue was introduced with the 27.x train' comment
Yes exactly, my thoughts as well. We tried that in Feb. We have had our case open since last year.
Pretty poor to release a product with such a major bug. Ive had more luck out of a dlink AP.
Are you running Meraki Switches?
I am running 28.3 firmware if tried every firmware release with a hope it will include the final fix.
We have four networks and with the exception of one, all have Meraki switches. But the issue is present in all of our networks.
Latest advice from Meraki support is to disable client balancing in one of our networks. Despite my willingness to upgrade the firmware to v28.x, they did not recommend it (?). The issue has now been escalated to their development team.
One last thing; most of our devices are Mac laptops. I haven't been able to verify the issue with Windows machines. I wonder this has to do with the NIC firmware vs Meraki AP.
Almost 6 months later and we are still waiting, despite weeks of chasing.
We are currently running the latest firmware but still having to reboot regularly to get decent performance.
Last update from Meraki..
Thank you for contacting Cisco Meraki Technical Support. Thank you for your email. I would like to inform you that our Dev Engineers are working on this issue. I will reach out to you once I have any updates. Thank you for your patience
Sigh.. roll on 2022!
I am having the exact same problem with a client of mine running MR52s. We are waiting on the next release for later this month. They say they hope to have a fix for this. 28.5 seems to be running better these days. We stayed on 27.7.1 for a while just to be sure, but recently upgraded more sites to 28.5. Fingers crossed.
Ok I have read through all the posts. Quick question What type of power are you hitting the APs with? If they are getting 802.3af power they will work but become way more persnickety and your dedicated scanning radio goes away and they become quite sticky from a channel standpoint. Also, the MR 32 is only capable of running the 26 family of FW.
Models with a dedicated scanning radio, such as the MR18/32/34/42/52/53/84, don't generate channel_scan events. They rely on the scanning radio to get the neighbor and channel utilization report on an ongoing basis.
In comparison, models without the dedicated scanning radio, such as the MR12/16/20/24/70, scan the whole spectrum every 2 hours when there are no clients associated. When low power mode is enabled the scanning radio function becomes disabled. Thus, not only are there no channel_scan events but AutoRF channel assignment can be negatively impacted.
We have had a very similar issue. Running approximately 60 MR52s across the enterprise. Tuning has been done manually for channels and power as found we could get a much better result than allowing the APs to manually complete this. Everything worked perfectly. my.meraki gave high download speeds consistently.
However after a few weeks to a month the usual report is poor performance and running a my.meraki speedtest which only tests bandwidth between the device and the AP shows a maximum speed of 20Mbs. We have tried to change the channels in use, power settings, channel width all to no avail. In some cases I have seen 0-1Mbs on the my.meraki test.
We have tried all the different Meraki firmware available and custom firmware, still to no avail. A ticket has been open with Meraki for months and still its in their "research and development section" with no solid answer forth coming.
I feel Meraki have really dropped the ball on this and they have had countless options to resolve it. I used to really believe in Meraki hardware, and while all devices have issues, the real key to customer service is how you deal with the problem. There is no way a retail shop would have got away with this fiasco.
We feel the Meraki devices have a hardware issue as I never saw these issues on MR33s or lower spec devices in the past. The symptom is a memory leak.
We also have a few MR56s we have installed and touch wood after weeks and weeks we have no issues at all so far.
These are affecting not only one office but multiple offices around as well, so it is not an environmental issue either as all offices are different types, some very isolated to the outside world so no chance of interference.
I have spent hours and hours with Meraki on this one and sent countless packet capture logs we had to acquire an Apple MAC so we could obtain the details they required.
However as of today I have just had to reboot 6 APs to resolve issues for clients and suspect I will need to do more.
I have even raised to Meraki they need to own this issue and arrange a scheduled staggered reboot to provide a stop gap for us. However that was met with a solid no. Also there is no uptime in the dashboard which would help predict the issue and be proactive. Meraki have uptime on their dashboard but customers dont...go figure...
We have also been told we are the only ones experiencing this issue, which I know is not the case, however this is the most recent post I have seen concerning these devices.
All I can suggest is push Meraki hard. There are issues as we are experiencing them as well.
We are also having the same problem. Smaller scale we have 4 MR42s
I had 2 of the MR42s replaced via RMA with Meraki.
I got about 4 weeks out of the new MR42s before the problem comes back.
It is not as severe as we get about 20-40Mbps on the my.meraki.com speedtest
But with my MR33s I'm getting 285-385Mbps up and down.
I have also been working on this issue for many hours with Meraki.
I bought Meraki to avoid issues like this.
There is no way to even schedule a reboot as far as I know.
At this point I would reboot them once per week on the weekend.
I'm a bit concerned in pushing the new 28.4 beta as I've read a post that it can cause some kinda of reboot loop.
@Emre666 we have Windows laptops, iOS devices and IoT devices with pretty much no Mac laptops, so you might have something there. Also I'm pretty sure we disabled load balancing years ago, but I can't find the settings right now to confirm!
The issue is intermittent and most of our Windows users are on the wired network. It might be the reason why our Win users haven't reported anything.
The client balancing is in your RF profile settings: Wireless > Radio settings > RF profiles
@Emre666 thanks, I was looking there and it was so obvious that I completely missed it! We actually have it enabled at the site I was at yesterday where everything is working well, so I'm not sure that can be the issue, but still worth trying as we did have to disable it in the past to improve client roaming.
I suspect that has to do with the issue either. We are experiencing the issue even when clients are not roaming. I firmly believe this has to do with the firmware since the issue started to happen around the same time (October 2020) in all of our networks. We did rollback the firmware to the version prior to that date but that did not help either. Go figure.
Yep, no change here either with Beta 28.4 - I still have the same nine MR42 APs that reduce to around 20Mb until they are restarted. They then run at full expected speed until whatever happens to reduce them to around 20Mb. Rinse and repeat.
So we've had this issue at both of our sites now for nearly 2 years. Meraki have been pretty useless at fixing the issue. Indeed it took forever for then to even acknowledge that this is a problem. We're running 28.5 and still no fix.
We're moving one of our sites over to Aruba because it's become very sensitive in terms of performance and Meraki simply have no fix. Indeed they are obviously hoping that the depreciation of the affected AP's will bury the problem.
We've been through every conceivable test, configuration change, update and the problem remains (AP's throttle WiFi down to about 20Mbs down 40-50Mbs up until rebooted, wired LAN tests are wirespeed.). We think you can force the issue by downloading a large amount of data 40-50GB but other than that it's appears to be quite random.
I'm stuck with them on my other site for about another year but we will be swapping them out for Aruba.
Meraki are in a bad place with this so do yourself a favour and start looking for your exit strategy.
I work with a network that has over 40 locations and we have experience the same issue. The AP speed goes down to 20Mbps. After a restart of the AP we get around 100Mbps. Our MR 42s are running version 28.5
An office with 10 MR52s. Speed has been at 20Mbps for a long time. Upgraded to 28.5 last night. This morning we saw speeds like 350Mbps. 12 hours after upgrade, speeds are not at 350Mbps anymore, but then again more people are in the office now. I will report back after a few days with an updated status.
You would likely have found a reboot would also bring the speeds back to what they should be as well. It will drop again in under a month most likely
Well after about a month now looks like our IT AP (the one I connect to and test on) is back to getting poor speeds. I was testing almost daily and getting 100+ Mbps speedtests from local status page. Currently getting 22 Mbps. Back to rebooting AP's. Lovely. This building has 201 MR53 AP's running 28.5.
I've been reading these forums for years and finally decided to make an account because of this thread. I have over 600 Meraki AP's (MR32, MR33, MR42, MR46, MR53, MR74). I'm in a school district and randomly get tickets or emails from teachers saying "internet is down or slow". We have one AP per classroom. A few weeks ago I was in a classroom with an MR42 and maybe 5 wireless devices connected. I went to the ap.meraki.com page and did a speedtest. I was getting 0.23 Mbps! Ping latency was very high. Checked the switches connected to AP and everything looks ok. Speeds were ok on another Meraki AP next classroom over. Upon reboot was now getting 136.02 Mbps. I've noticed this with MR42 and MR53 AP's. Running firmware 27.7.1. Next time this is happening I'm going to open a case with Meraki.
So this appears to be the exact scenario that users will run into and as much as I hate to say it, I do not think Meraki will ever be able to find a fix for this. It's been on-going for years. Support will do what they can, but at the end of the day they are going to ask you to upgrade firmware. That is the only solution they have. They never seem to be able to actually get to the root of what causes the performance to go to null, only that changing the channel on the radio, or rebooting the AP, will fix it.
This is why I have automated scripts that reboot every AP in every network (at specific times) once a week. I have over 2500 AP's. This is the only solution I could find (at least for me) to be able to get in front of it so that I don't get those complaints. After a case was opened for over 1.5 years with the only solution being upgrade firmware over and over again, I had to do something. My next step is out the door, to a land more misty.
Curious which model Meraki AP's you have? Weird thing is that I've not seen the issue at all on MR33 or MR46's. I'm hoping to get a few MR56's in soon. Moving to another platform like Aruba has been discussed internally.
Can't say for certain to be honest, too many AP's to monitor. The one other issue I know I ran into was on MR32/MR33 (at the time we had nothing else) the 5GHz radio would stop accepting clients, but still broadcast beacon frames, thus creating a black-hole. That was another endless 'upgrade firmware' solution. Only fix was to also change channel or reboot AP.
I thought MR56s were not susceptible to this issue, however we are seeing some weird issues with these models again where a reboot once again resolves it. Have not bothered to raise any more cases as they just end up in "development" team excuses again.
@zzzthumpI have an MR56 that hasn't been rebooted in a while and haven't seen this issue. We also have a lot og MR42/52 models and also haven't had the kind of issues experienced here. I wonder if there is a particular combination of configurations that cause it?
What authentication do you use?
What channel width?
Access lists on APs?
There has to be something!
Hi. Unsure what is causing it. Seems to be a lot experiencing the issues but advised there is no issue....hmmmm
40-80MHz, has not made a difference.
Band Steering Yes
Client balancing is off
We tend not to use band steering and at most sites stick to 20MHz but I just checked and one of our busiest sites does have band steering on the SSID used for laptops where the performance would be most quickly noticed. Otherwise the same as you.
I wouldn’t bother wasting your time changing/playing with your config/setup. We went through every conceivable setup and testing and nothing worked. We’ve had this problem for about 2 years now and the only ‘fix’ is to reboot them (we also use python scripts to do this overnight). We’re currently running firmware 28.5 which hasn’t helped. They appear unable to fix this. I have already made arrangements to replace them on one site with Aruba. I will be making arrangements after the for the other site. If they can’t fix it in 2 years I can’t see they are ever going to. They probably won’t officially acknowledge this as the RMA cost would be astronomical.
So I asked our account manger to swap out the MR42's we have for models that aren't affected and he's been told they won't do it FoC. So no swap out and no fix. No even a real acknowledgement of the issue. For 2 years we have been dealing with issue. For those joining this tread let me advise you that no amount of playing with configs will change this. We've been through every conceivable config change/testing/setup. Only solution seems to be replace Meraki with another vendor which is what I intend to do when budgets allow.
How many access points are we talking @CarlMaycock ?
It's my humble opinion that this is terrible customer support. The hardware you have clearly has an issue, they can see it themselves, but are not able to resolve it (software/config) and are unwilling to replace. At the very least they could swap out like (5) of your MR42's and replace them with something else in a corner of your building, so that they can get their hands on your hardware, and test to see what happens.
@MeredithW is there someone you can flag because this seems like the account manager is not doing his job.
We have about 170 over 2 sites.It seem crazy doesn’t it? I guess our account manager won’t have the authority on it. It’ll be higher up the scale. I suspect the reason is that they’d be hit with a tsunami of replacements (see all above) and associated costs.
So after cornering my account manager Meraki confirmed that they wouldn't swap out the AP's due to cost. So I asked them what the plan was as I've had a service affecting bug for over 2 years now that they can't/won't fix and they won't swap out the AP's for models that aren't affected. I was advised by my account manager that they are planning on 'fixing' the issue in firmware version 28.6 however this is not 100% confirmed. I'm not holding my breath. I will ,however, be making plans within my next budget for replacing Meraki with Aruba as I've already done at another site.
I suspect that there are a huge amount of sites that doesn't even realise they have this issue as most end users will only be using them for light internet access.
My feeling is that they can't 'fix' this as I suspect there is a chip somewhere that's is being throttled probably due to thermal issues. If that is the case it's unlikely a firmware fix will do it. Certainly 2 years in would suggest it's a problem they can't fix. Having been through multiple firmware updates that didn't fix it I have no reason to think yet another will.
We have fought for over a year on this with promised updates and firmware fixes. We now suspect its a major issue at the hardware level and resources are being depleted until it is rebooted as you say. Meraki are not interested in replacing the APs with MR56s which dont seem to have the same issue.
We have close to 100 APs suffering from this issue. We have decided to dump Meraki once and for all. We are migrating to the MIST platform. More expensive but after 3 months trial its still performing perfectly with advanced metrics and troubleshooting ability. Meraki used to be top shelf stuff, however there is better now in the Gartner quadrants. Good luck. Cheers
Huge news. My case number for those interested is 07376291 - Throughput and performance issues on APs. We took packet captures and logs on AP (specifically MR53) before and after reboot. There is an internal bug case with Meraki since April 2021. They started developing a fix in August 2021. There is a pre beta release 28.6 that is now available and I'm testing with on an AP in our IT Office. This new custom firmware became available January 13, 2022. Apparently, the issue is fixed. AP Models affected are MR42, MR42E, MR52, MR53, MR53E, MR84. I had a feeling this was the case since I wasn't seeing the issue on our MR32 or MR33 AP's. Fingers crossed!
We recently started having issues with our MR42's where they would have to be restarted daily and sometimes twice daily as the speed just kept dropping off.
We spoke with Meraki yesterday and also found out about this non-released BETA version that we are trying on 3 of our AP's that were most affected. We are also hoping that they have resolved the issue in this release, but they said that about the previous two!
I'll keep my fingers crossed, but I won't hold my breath!
Stable release candidate MR 28.6 released today. Discussed with support and this specific bug wasn't mentioned in the changelog which annoys me. They assured me it was fixed. I just updated 201 AP's in my High School.
@Macguy is the below fix not the one for the issue in this thread:
I originally thought so. The MR33 is technically a wifi 5 wave 2 AP and isn't affected by the bug (per support). I asked if this ADDBA request bug fix was related to this case and they weren't sure. All I got was the following:
Advised customer that from the internal issue tracking information, NSE is able to confirm that the development team mentioned the fix is included in the MR28.6 firmware version.
So far so good here on my end with 28.6. Haven't seen the degraded performance over time. There was a recent post in the Meraki reddit thread here:
Yep, we've had 28.6 running at one site for a few weeks now and it appears stable. We've upgarded another site and are monitoring. Hopefully this will resolve the problem, albeit very late on in the day.
We upgraded all the sites at our affected client with this firmware as well and are removing scheduled reboots to test over the next month.
Well it's been over a month and I've tested this on MR53 and MR42 AP's... no degraded performance. I'm considering this resolved with 28.6. Unfortunately, it took Meraki two years to fix LOL.
We are now running 28.6 at both sites and it appears to be holding up. No evidence of throttling as of yet. Interestingly there's still no official acknowledgement of the issue or fix beyond support advising that 28.6 fixes the 'non-acknowledged' problem 🙂