MR42/MR52 bad performance

Marc_V
Comes here often

MR42/MR52 bad performance

Hello,

 

We have replaced our MR32 for a mix of MR42/MR52 and since this change, we can see that performance (bandwidth) slowly getting lower and lower and almost tend to less than 1 Mbps. Only a reboot of the device permit to have good performance back for some time.

We are observing this behavior in environment where there is multiple other SSIDs broadcasted by other third parties in the buildings next to us.

We are in 5GHz only, 40 Mhz width channel. The 5 Ghz Channels used are not overloaded in utilization by other third parties.

If i put a MR32 at the same place, same channels, i don't have the issue.

It's like MR42/MR52 have an issue when the environment is used a lot even if it is not on the channels we are using. And, a reboot solve the issue for some hours (and we keep the same channels, so it's not a question of channels overloaded).

We have open ticket to support but upgrading to last firmware does not help.

 

Thanks in advance.

53 REPLIES 53
PhilipDAth
Kind of a big deal

Try setting the minimum connect speed to 12Mb/s (unless you are using something higher already).  Perhaps legacy protocols are consuming a lot of airtime, or devices are holding onto APs some distance away.

 

Also note it takes 24 hours before the final channel plan and power levels gets put into affect.  The APs slowly drift towards their final values.

https://documentation.meraki.com/MR/Monitoring_and_Reporting/Location_Analytics/Meraki_Auto_RF%3A__W... 

 

If you do a speedtest between the AP and the user (using my.meraki.com) is the result also poor?

 

Are you using the 26.x firmware?

Hi Philip,

 

Thanks for your answer.

I am already with minimum connect speed 12 Mb/s. 

When the problem occurs, the speedtest using my.meraki.com show a result between 0 and 1 Mbps. So users can't do anything in Wifi.

We are using the 26.6.1 firmware, we had the same issue in 25.13.

This behavior is seen only on MR42/MR52 access point. I don't have this issue with MR32 access point. Seems something related to MR42/MR52 model

My only way to solve it temporarily is to reboot the access point.

NolanHerring
Kind of a big deal

I would try 26.7, change to 20MHz channels (just in case). What is your channel utilization being reported as when you run those tests to the AP directly and its only giving 1Mbps?
Nolan Herring | nolanwifi.com
TwitterLinkedIn
cmr
Kind of a big deal
Kind of a big deal

Also enable DFS channels unless you are near a radar user.  Without DFS in Europe there are only two 5GHz channels, so the interference can slow them right down.

Marc_V
Comes here often

Hello,

Thanks for your answers guys !


Regarding the channel utilization, it is always pretty fine on the channels used by the access points (40 Mhz), so it's seems not to be related to a high utilization of the channels currently in use by the access point itself.And if i replace the MR42 or MR52 with a MR32 with the same channels, i don't have the issue.

 

So it's seems related only to these 802.11ac wave 2 models and not especially if the channels used are highly used but also if all the channels around are high used (even if not overlapping with the channels currently in use).


I was not able to test many things since lockdown but i have seen that in the release notes of the 26.8 firmware, an issue fixed can correspond to my problems:
-> MR would get into a state where it would cease being able to TX frames reliably in environments with sustained high channel utilization (802.11ac Wave 2 MRs)

This corresponds to the fact that my issues are only seen on these MR42 and MR52 (802.11ac wave 2) and not on old MR32.

I will upgrade and check how it goes.

 

Thanks !

Hi,

 

Did you ever make any headway with this issue?

 

We have a lot of MR42's and are now also seeing very poor performance with them until they are rebooted.


We have checked out the switching environment, check our WAN usage, and have gone over our RF profiles and cannot see the cause of the issues.

 

Like yourself, rebooting the AP returns it back to good performance, but with 180 across our site, doing this regularly with what is supposed to be enterprise hardware is not ideal!

 

We are running the latest firmware as well.

 

Kind regards


Matt

cmr
Kind of a big deal
Kind of a big deal

Hi @TASIS_IT , when you say the latest, can you be more specific, there is 27.6 (which I guess you mean), but others out there as well?

Hi @cmr,

 

Yes, we are running 27.6, across all of our MR42's

 

Nothing else is currently available on the Merkai Portal.

 

Kind regards

 

Matt

Hi Matt

 

Check my post in this thread if you are still having issues.

 

Cheers

 

Michael

 

Hi All,

 

After weeks of sending screenshots and packet captures, I received this response from Meraki.

 

I have managed to find the issue that is causing the slow speed problem over time and after a reboot, the issue just goes.
Thanks to the packet capture taken from the working and AP and broken AP we identified that the issue was being caused because the client was sending aggregated frames and the AP wasn't.

Initially, both the AP and the client decide to send aggregate frames then all of a sudden after a while the AP doesn't send aggregate frames. Hence when you restart the AP it's all fresh and negotiates with the client again to send aggregate frames and the problem disappears.

We are currently working, on fixing this issue on a feature firmware release with our internal team, however, at this point, the problem is still premature and we are looking more into why the AP goes into stops sending aggregate frames. We know that the issue came around firmware version 27.x. Our internal team has asked me to ask you if you would be willing to test our beta MR 28.1 to see if the problem also persists on this firmware.

If you are willing to upgrade a couple of APs to MR28.1 to test let me know and I can arrange a session to ping the firmware on the APs you want to test on without having to upgrade the whole network to MR28.1

 

I am just waiting for our summer holidays to start and then I will be in a position to test MR28.1 on a few of the AP's.


I was pleased to see that they had identified a possible source of the issue since we have been chasing this for weeks and questioning ourselves whether it was something that we had done or something we were missing with the configuration of our network.

 

Hopefully, the issue is resolved in the next firmware update, but ill update you all again once i have tested further.

 

Kind regards


Matt

cmr
Kind of a big deal
Kind of a big deal

Hi @TASIS_IT,

 

We are running 28.1 on most of our 100+ APs and at least 30 of those are MR42/52 models.

 

It has been stable so far, I'll let you know if we see any problems.

Hi @cmr 

 

Do you still have any performance issues with v28.1 these days?

Thank you.

cmr
Kind of a big deal
Kind of a big deal

@HMP we are on 28.2 now and that seems fine.  I will aim to do some more testing tomorrow to confirm.

zzzthump
Comes here often

How long have you been on 28.2? We moved to 28.1 and then 28.2 

 

However I have just had exactly the same issue out of firmware version 28.1 . Slow speed between AP and client verifying with my.meraki.com and doing the speed test.

It was upgraded June 22nd, and today well its July 15th. 

For an expensive AP it lasts less time than some cheap AP. 

 

Also I have noticed on the latest MR56 after a few months its dumping clients off trying to connect it. There seems to be no light at the end of the tunnel at all with these APs.

 

 

 

Meraki, you really need to sort this out. 

 

 

 

zzzthump
Comes here often

Ours has just fallen over again on V28.1

 

We got about 3 weeks out of them again. Useless

 

 

 

Emre666
Comes here often

Hi @zzzthump,

 

Has the beta firmware improved anything? I just noticed there is a 28.4 beta release and I am thinking of trying that.

 

We have a total of 22 MR42s and have been dealing with the exact issue for almost a year now. Super slow network and a lot of 802.11 disassociations with "unknown error", where clients are disconnected from the network.

 

Emre

You could always downgrade to 26.8.1

Nolan Herring | nolanwifi.com
TwitterLinkedIn

Hi


We have tried that version. Still had issues.

If that is the case, that sort of pokes a hole in the 'this issue was introduced with the 27.x train' comment

Nolan Herring | nolanwifi.com
TwitterLinkedIn

Yes exactly, my thoughts as well. We tried that in Feb. We have had our case open since last year.

 

Pretty poor to release a product with such a major bug. Ive had more luck out of a dlink AP.

 

Are you running Meraki Switches?

I am running 28.3 firmware if tried every firmware release with a hope it will include the final fix.

 

cmr
Kind of a big deal
Kind of a big deal

Just to reassure you on the reboot loop, we have over 50 MR42s running release 28.4 and we don't have the reboot issue.

Emre666
Comes here often

We have four networks and with the exception of one, all have Meraki switches. But the issue is present in all of our networks.

 

Latest advice from Meraki support is to disable client balancing in one of our networks. Despite my willingness to upgrade the firmware to v28.x, they did not recommend it (?). The issue has now been escalated to their development team.

 

One last thing; most of our devices are Mac laptops. I haven't been able to verify the issue with Windows machines. I wonder this has to do with the NIC firmware vs Meraki AP.

 

TBHPTL
Getting noticed

Ok I have read through all the posts. Quick question What type of power are you hitting the APs with? If they are getting 802.3af power they will work but become way more persnickety and your dedicated scanning radio goes away and they become quite sticky from a channel standpoint. Also, the MR 32 is only capable of running the 26 family of FW.

https://documentation.meraki.com/General_Administration/Firmware_Upgrades/Product_Firmware_Version_R...

 

 

https://documentation.meraki.com/MR/Monitoring_and_Reporting/Low_Power_Mode

 

 

Models with a dedicated scanning radio, such as the MR18/32/34/42/52/53/84, don't generate channel_scan events. They rely on the scanning radio to get the neighbor and channel utilization report on an ongoing basis.

 

In comparison, models without the dedicated scanning radio, such as the MR12/16/20/24/70, scan the whole spectrum every 2 hours when there are no clients associated. When low power mode is enabled the scanning radio function becomes disabled. Thus, not only are there no channel_scan events but AutoRF channel assignment can be negatively impacted.

zzzthump
Comes here often

Hi

 

We have had a very similar issue. Running approximately 60 MR52s across the enterprise. Tuning has been done manually for channels and power as found we could get a much better result than allowing the APs to manually complete this. Everything worked perfectly. my.meraki gave high download speeds consistently. 

 

However after a few weeks to a month the usual report is poor performance and running a my.meraki speedtest which only tests bandwidth between the device and the AP shows a maximum speed of 20Mbs. We have tried to change the channels in use, power settings, channel width all to no avail. In some cases I have seen 0-1Mbs on the my.meraki test.

 

We have tried all the different Meraki firmware available and custom firmware, still to no avail. A ticket has been open with Meraki for months and still its in their "research and development section" with no solid answer forth coming. 

 

I feel Meraki have really dropped the ball on this and they have had countless options to resolve it. I used to really believe in Meraki hardware, and while all devices have issues, the real key to customer service is how you deal with the problem. There is no way a retail shop would have got away with this fiasco.

 

We feel the Meraki devices have a hardware issue as I never saw these issues on MR33s or lower spec devices in the past. The symptom is a memory leak.

 

We also have a few MR56s we have installed and touch wood after weeks and weeks we have no issues at all so far. 

 

These are affecting not only one office but multiple offices around as well, so it is not an environmental issue either as all offices are different types, some very isolated to the outside world so no chance of interference.

 

I have spent hours and hours with Meraki on this one and sent countless packet capture logs we had to acquire an Apple MAC so we could obtain the details they required. 

 

However as of today I have just had to reboot 6 APs to resolve issues for clients and suspect I will need to do more. 

 

I have even raised to Meraki they need to own this issue and arrange a scheduled staggered reboot to provide a stop gap for us. However that was met with a solid no. Also there is no uptime in the dashboard which would help predict the issue and be proactive. Meraki have uptime on their dashboard but customers dont...go figure... 

 

We have also been told we are the only ones experiencing this issue, which I know is not the case, however this is the most recent post I have seen concerning these devices.

All I can suggest is push Meraki hard. There are issues as we are experiencing them as well.

 

 

 

 

Cheers

 

 

Michael

 

zzzthump_0-1624227907963.png

 

 

zzzthump_0-1624226923542.png

 

 

We are also having the same problem. Smaller scale we have 4 MR42s

I had 2 of the MR42s replaced via RMA with Meraki.

I got about 4 weeks out of the new MR42s before the problem comes back.

It is not as severe as we get about 20-40Mbps on the my.meraki.com speedtest 

But with my MR33s I'm getting 285-385Mbps up and down.

 

I have also been working on this issue for many hours with Meraki.

I bought Meraki to avoid issues like this.

 

There is no way to even schedule a reboot as far as I know.

At this point I would reboot them once per week on the weekend.

 

I'm a bit concerned in pushing the new 28.4 beta as I've read a post that it can cause some kinda of reboot loop.

Emre666
Comes here often

I've tried reboot a few times to no avail. Does reboot temporarily fix the issue?

cmr
Kind of a big deal
Kind of a big deal

@Emre666 we have Windows laptops, iOS devices and IoT devices with pretty much no Mac laptops, so you might have something there.  Also I'm pretty sure we disabled load balancing years ago, but I can't find the settings right now to confirm!

Emre666
Comes here often

The issue is intermittent and most of our Windows users are on the wired network. It might be the reason why our Win users haven't reported anything.

 

The client balancing is in your RF profile settings: Wireless > Radio settings > RF profiles

cmr
Kind of a big deal
Kind of a big deal

@Emre666 thanks, I was looking there and it was so obvious that I completely missed it!  We actually have it enabled at the site I was at yesterday where everything is working well, so I'm not sure that can be the issue, but still worth trying as we did have to disable it in the past to improve client roaming.

Emre666
Comes here often

I suspect that has to do with the issue either. We are experiencing the issue even when clients are not roaming. I firmly believe this has to do with the firmware since the issue started to happen around the same time (October 2020) in all of our networks. We did rollback the firmware to the version prior to that date but that did not help either. Go figure.

Hi cmr,

 

It seems like Beta version 28.4 still got poor performance.

Yep, no change here either with Beta 28.4 - I still have the same nine MR42 APs that reduce to around 20Mb until they are restarted. They then run at full expected speed until whatever happens to reduce them to around 20Mb.  Rinse and repeat.

 

 

CarlMaycock
Conversationalist

So we've had this issue at both of our sites now for nearly 2 years. Meraki have been pretty useless at fixing the issue. Indeed it took forever for then to even acknowledge that this is a problem. We're running 28.5 and still no fix.

 

We're moving one of our sites over to Aruba because it's become very sensitive in terms of performance and Meraki simply have no fix. Indeed they are obviously hoping that the depreciation of the affected AP's will bury the problem.

 

We've been through every conceivable test, configuration change, update and the problem remains (AP's throttle WiFi down to about 20Mbs down 40-50Mbs up until rebooted, wired LAN tests are wirespeed.). We think you can force the issue by downloading a large amount of data 40-50GB but other than that it's appears to be quite random. 

 

I'm stuck with them on my other site for about another year but we will be swapping them out for Aruba.

 

Meraki are in a bad place with this so do yourself a favour and start looking for your exit strategy.

@CarlMaycockAre you using MR42s across the board?

Yes we have about 150 MR42's and about 20 MR72's (these are unaffected).

 

 

Audry
Comes here often

I work with a network that has over 40 locations and we have experience the same issue. The AP speed goes down to 20Mbps. After a restart of the AP we get around 100Mbps. Our MR 42s are running version 28.5

rkausland
Comes here often

An office with 10 MR52s. Speed has been at 20Mbps for a long time. Upgraded to 28.5 last night. This morning we saw speeds like 350Mbps. 12 hours after upgrade, speeds are not at 350Mbps anymore, but then again more people are in the office now. I will report back after a few days with an updated status. 

You would likely have found a reboot would also bring the speeds back to what they should be as well. It will drop again in under a month most likely

Macguy
Comes here often

I've been reading these forums for years and finally decided to make an account because of this thread.  I have over 600 Meraki AP's (MR32, MR33, MR42, MR46, MR53, MR74).  I'm in a school district and randomly get tickets or emails from teachers saying "internet is down or slow".  We have one AP per classroom.  A few weeks ago I was in a classroom with an MR42 and maybe 5 wireless devices connected.  I went to the ap.meraki.com page and did a speedtest.  I was getting 0.23 Mbps!  Ping latency was very high.  Checked the switches connected to AP and everything looks ok.  Speeds were ok on another Meraki AP next classroom over.  Upon reboot was now getting 136.02 Mbps.  I've noticed this with MR42 and MR53 AP's.  Running firmware 27.7.1.  Next time this is happening I'm going to open a case with Meraki. 

NolanHerring
Kind of a big deal

 

So this appears to be the exact scenario that users will run into and as much as I hate to say it, I do not think Meraki will ever be able to find a fix for this. It's been on-going for years. Support will do what they can, but at the end of the day they are going to ask you to upgrade firmware. That is the only solution they have. They never seem to be able to actually get to the root of what causes the performance to go to null, only that changing the channel on the radio, or rebooting the AP, will fix it.

This is why I have automated scripts that reboot every AP in every network (at specific times) once a week. I have over 2500 AP's. This is the only solution I could find (at least for me) to be able to get in front of it so that I don't get those complaints. After a case was opened for over 1.5 years with the only solution being upgrade firmware over and over again, I had to do something.  My next step is out the door, to a land more misty.

Nolan Herring | nolanwifi.com
TwitterLinkedIn

Hey Nolan,

 

Curious which model Meraki AP's you have?  Weird thing is that I've not seen the issue at all on MR33 or MR46's.  I'm hoping to get a few MR56's in soon.  Moving to another platform like Aruba has been discussed internally. 

NolanHerring
Kind of a big deal

Mix of MR32/MR33/MR42/MR72/MR74/MR84/MR36/MR44

Nolan Herring | nolanwifi.com
TwitterLinkedIn

You've seen this issue on the MR33's?  None of my three elementary buildings have reported issues (all MR33). 

NolanHerring
Kind of a big deal

Can't say for certain to be honest, too many AP's to monitor. The one other issue I know I ran into was on MR32/MR33 (at the time we had nothing else) the 5GHz radio would stop accepting clients, but still broadcast beacon frames, thus creating a black-hole. That was another endless 'upgrade firmware' solution. Only fix was to also change channel or reboot AP.

Nolan Herring | nolanwifi.com
TwitterLinkedIn
zzzthump
Comes here often

I thought MR56s were not susceptible to this issue, however we are seeing some weird issues with these models again where a reboot once again resolves it. Have not bothered to raise any more cases as they just end up in "development" team excuses again. 

cmr
Kind of a big deal
Kind of a big deal

@zzzthumpI have an MR56 that hasn't been rebooted in a while and haven't seen this issue.  We also have a lot og MR42/52 models and also haven't had the kind of issues experienced here.  I wonder if there is a particular combination of configurations that cause it?

 

What authentication do you use?

What channel width?

Access lists on APs?

Band steering?

 

There has to be something!

zzzthump
Comes here often

Hi. Unsure what is causing it. Seems to be a lot experiencing the issues but advised there is no issue....hmmmm

 

WPA2 

40-80MHz, has not made a difference.

No ACLs

Band Steering Yes

Client balancing is off

 

cmr
Kind of a big deal
Kind of a big deal

We tend not to use band steering and at most sites stick to 20MHz but I just checked and one of our busiest sites does have band steering on the SSID used for laptops where the performance would be most quickly noticed.  Otherwise the same as you.

I wouldn’t bother wasting your time changing/playing with your config/setup. We went through every conceivable setup and testing and nothing worked. We’ve had this problem for about 2 years now and the only ‘fix’ is to reboot them (we also use python scripts to do this overnight). We’re currently running firmware 28.5 which hasn’t helped. They appear unable to fix this. I have already made arrangements to replace them on one site with Aruba. I will be making arrangements after the for the other site. If they can’t fix it in 2 years I can’t see they are ever going to. They probably won’t officially acknowledge this as the RMA cost would be astronomical. 

Macguy
Comes here often

WPA2

40 MHz

No ACL

No Band Steering

No Client Balancing

nb12345
Here to help

Anyone had any luck resolving this? I am also experiencing the same issue on our mr42’s

Eduard_Medla
New here

Hello,
one our big client, with thousands of APs of such a type is affected. Please resolve this asap.
Thank you

Get notified when there are additional replies to this discussion.
Welcome to the Meraki Community!
To start contributing, simply sign in with your Cisco account. If you don't yet have a Cisco account, you can sign up.