WLAN speeds suddenly became erratic

[RumorConsumer]

I have an MX65 that is connected to a network of around 10 gateways and a few repeaters. 

One of the gateways is an MR52 which is connected directly to the MX65's 2nd POE+ port. 

My WAN is a ATT Enterprise fiber 250/250 connection that has been flawless. Recently, speeds have been getting weird from wireless clients. As I always point out in my posts, I am in a very rural area with no other RF interference (its a retreat center). I shape the bandwidth on the SSID in question (the only one being used out of 3 - we have very vey light usage) as follows:

This has been solid, great no problems. Enough speed that somebody doing something important gets a decent rate, and speed burst is always a welcome bonus. I have changed nothing in the config, but recently people have been complaining that things have been slow. I didnt believe them until today I had somebody run a speed test from this particular AP and sure enough, to multiple endpoints at major telecoms where the speeds used to be perfect, I see 60 down/224 up, 70 down/225 up etc etc. So wildly different speeds, and nowhere within my set parameters for shaping, and way slow for download. On another gateway AP, this does not occur. 

What could be the problem here? Could this be defective hardware? The only other thing Ive done is update firmwares on both the MX and the APs to all the latest betas. I could try a downgrade but why would that matter if other APs are working properly? 

Update - in terms of trends, the only other AP we know of that is having a problem like this is another MR52. The wired AP that is just fine is an MR42. I wonder if there is a recent bug in the firmware for MR52s... I have several others wired on campus and will try them to see if theyre also affected.

Update 2 - Ok its not MR52 exclusive. I have tried another one, wired, in a different part of my network and its performing great... except its *over* speed. Its 214/214 or so which is wrong. So it looks like my bandwidth shaping is f'd. Any other weigh ins? 

Update 3- I removed all bandwidth shaping and had them try it again. Pretty much the same results. its the most I could do while everybody was still online and its still present. Downloads are at most about 100mbit on this odd assortment of APs. I will roll the firmware back to latest stable release candidates overnight and see if that helps. 

Update 4- I was relieved to learn that it was the middle of the night for @PhilipDAth when I posted this 🙂 I was concerned when I didnt get a reply within 5 minutes 🙂 

Update 5 - I rolled back the firmware on the MX and MRs back to safe versions - no change. Still super wonky but only on some APs. No pattern to which model. Happens on the internal POE+ port as well as downstream after 2 switches, but in other cases after several other switches other wired APs are just fine. What the heck?

[BlakeRichardson]

@RumorConsumer  I was going to say have you had any recent firmware upgrades however I saw that was the last thing you looked at. 

How are you measuring the speeds you are taking, are they LAN > WAN speeds or LAN >LAN?

[PhilipDAth]

Good point @BlakeRichardson .  Do you get an ok result using the built in client<->AP speed test (accessed using my.meraki.com)?

[RumorConsumer]

So when i use my.meraki.com i get proper throughput speeds. So we know at the link level all is well - 140mbit from a test just now at an affected AP. Back to the same question - why isn’t there actually internet bandwidth not flowing properly to those clients in those APs? Im using Speedtest app and site to determine speed to the net and have used at least 5 endpoints each time i test. Again it’s like 50-80 down on the affected units and on the ones that are working fine we’re taking the normal 170mbit or so. So weird. 

[rhbirkelund]

Have you applied a custom radio profile? How is it conifgured?

[RumorConsumer]

Nope. Nothing out of the basic. And verified the change log on the org - nada.

[rhbirkelund]

Copy the working radio profile, and move alle the APs to the new profile. Then tweak the new profile.

Assuming you haven't already, move the Minimum Bitrate slider to 12, to disallow all legacy bitrates, and see if that changes anything.
Make sure to set Band selection to Per AP in the Profile -> General section.

If you get more issues, you can just move the APs back to the original Radio Profile.

[RumorConsumer]

@rhbirkelund I could certainly try that, but the link level speed is confirmed as good via the my.meraki.com throughput test that tells us that the pipe from the AP to the client is operating normally, no? If the bitrate was somehow stuck low we would be see link-level issues but my problem appears to be higher up in the OSI hierarchy. 

[rhbirkelund]

I was just curious to see if could be something with the negotiated bitrate and modulation schemes jumping up and down.
Setting to minimum bitrate of 12, disables legacy modulation schemes which could induce some interference. However, this could also result in reduced coverage, since it will require a higher SNR.
On the other hand, allowing the lower bitrates will also make your Wifi coverage larger and more robust, enabling you to keep a connection with a lower SNR.
Pros and cons. 🙂

Have you tried setting up an iPerf server, and measure throughput that way?

[BlakeRichardson]

@RumorConsumer  you haven't setup any traffic shaping rules recently?

[RumorConsumer]

Zero. It does seem like my though right?

[RumorConsumer]

Just bumping this thread. I to reiterate, I havent changed anything. The org log reflects that so nobody else (my one other admin) hasn't tampered at all. And I can use ap.meraki.com and the link level speed test reports perfect bandwidth. So weird.

[RumorConsumer]

Shoot. As in I think I shot myself in the foot. I had implemented a new RF profile a few months ago, the only difference in which was to eliminate UNII-1 for a small corner of the property (the same corner affected LOL) bc I was going to use some other 5ghz equipment that occupied that space. Not doing that anymore but I left the profile in place. The profile was in place on 2 APs, one was working fine and the other was one of the symptomatic ones. I think I didnt realize that at 80mhz and eliminating UNII1 leaves precious little room to maneuver and would produce a situation where one AP was fine and the other was hobbled. I had figured if both weren't problematic then that wasn't it. Seems I was wrong. We reenabled UNII1 and moved to 40mhz channels (I think i could do 80 but we dont miss the speed and Id rather have the range) and the problem has disappeared. Thanks all for your usual weigh ins and props to @rhbirkelund who sniffed it out.

[cmr]

@RumorConsumer glad to see you found the issue, that does of course all make sense, I found some similar issues with trying 80MHz channels even with all bands enabled.  There seem to be a max of three channels and where I am that means one of mine has to overlap directly with one of the neighbours, so I too moved down to less, but cleaner, bandwidth.

[RumorConsumer]

Thanks @cmr 

The next thought I have is that 40mhz will actually work better if I need to cinch a few APs off from UNII-1 in the future. It will give twice as many channel options in the higher # channels if I need that space for a point to point link. 

[cmr]

Indeed, @RumorConsumer I think the best way to approach it is; do I need the extra possible bandwitdth or user density that the wider channels can give, if so can I afford it without overlapping channels.

[RumorConsumer]

@cmr yeah I dont think I do. 40mhz with multiple streams still clocks us in above 250mbit. Anything we are doing involving higher bandwidth would be done wired or over our 10Gbit fiber backbone. This is really just to make sure standard internet works well for standard/highly communicative internet users who want fast, low latency voice over data, Netflix, iMessage, web, software updates, etc. Nothing more really. Plus I limit bandwidth to 150mbit or so down so its really a non issue. 80mhz would just be a nerd flex for me I think hehe