WIFI dropping several times a day

[SchoolITMan]

We are having issues with teachers or students dropping off WIFI several times a day.  This inevitably happens in the middle of a Zoom class and is effectively hobbling our ability to do distance learning.

We are a mixed show with roughly 20% Windows PCs and 80% Macbook Air laptops.  We have a full Meraki network, using an MX250 firewall, MS250 Core switch, and a few dozen MS120 48 port switches.  The switches are on firmware 12.28   Our WIFI APs are MR42 units indoors and MR70 units outdoors.  The APs are on firmware 26.8.1 (recently downgraded from 27.5) We have 2 buildings, on the same network, connected by fiber.  Only one of the buildings is experiencing issues.

The Students are on an SSID with WPA2-PSK.  The Faculty and Staff are on a separate SSID that is using WPA2-Enterprise with Meraki RADUIS. (Using the built-in user auth, not AD)

Our logs are full of disassociation and authentication errors.  Our APs wireless health shows stats like 54% of clients unable to connect.  46% of authentication failed.  Drilling down shows mostly things like:

• "Client failed during the authentication step.type='WPA-PSK auth fail' associated='false' radio='1' vap='2'"
• "Client failed 802.1X authentication to the RADIUS server.type='802.1X auth fail' num_eap='8' first_time='0.018824960' associated='false' radio='1' vap='8'"
• "Client failed 802.1X authentication to the RADIUS server.type='802.1X auth fail' num_eap='13' first_time='0.049299200' associated='false' radio='1' vap='1'"
• "Client attempted to associate 46 times on the 5 GHz band.type='Association attempts' num='46' associated='false' radio='1' vap='8'"
• "Client attempted to associate 179 times on the 5 GHz band.type='Association attempts' num='179' associated='true' radio='1' vap='1'"

But no-one is typing in a password here.  These are saved credentials.  And 300+ people cant all have suddenly developed corrupt keychain entries on their Macbooks.

Can anyone tell me what is going on here?

[BrandonS]

I would start thinking about interference.  In part, because you mentioned this happens in one building and not another.  

Go to wireless > monitor > RF spectrum and see how things look.  

A couple possibilities might be, non wifi interference or rogue devices.  Rogue devices are pretty common in school environments in my experience.  Someone will be unhappy with wireless performance and decide they can make it better by bringing their own repeaters and other gear into your environment.  Also, portable hotspots via phone or individual devices can cause interference.

[SchoolITMan]

I've looked at the RF Spectrum, and it definitely does NOT look clean, but it doesn't look any different for "problem APs" as it does for those APs that are not having any issues.  The interfering APs are all under -75db and under -85db on the same 5hgz channel.  Same channel utilization is only 15% current and avg.

[cmr]

As one building is different to the other, what other differences might there be; client density, client type, AP density, AP no. of neighbours etc.?

[AjitKumar]

Hi @SchoolITMan 

As everyone is suggesting "interference" could be the primary reason.

Also seems like we have a new Firmware release - MR 27.5.1.

You may give it a shot - if already not upgraded.

https://community.meraki.com/t5/Wireless-LAN/New-MR-27-5-1-firmware/m-p/101264

[SchoolITMan]

 @ Ajitkumar

Yes, per the OP, we just downgraded from 27.5.1 in troubleshooting this problem.  It was actually worse on 27.5.1.

@ cmr

Yes, student and classroom density is higher in this building than in the other.  As is general usage.  But this was working fine for years until Sept 28th.  No new businesses or construction near us, we own most of the land around us.  I can't imagine that interference got this much worse all of a sudden.

What else would cause repeated Authentication issues?

[cmr]

@SchoolITMan what was it that caused the change, upgrading to 27.x, or something else in the environment?

[SchoolITMan]

Not sure.  About that time (give or take a week or two), we:

• We added several new MR70 outdoor WIFI units (only 1 at the affected campus)
• We were having issues with our MX250, and we upgraded the firmware on the MX, the Switches and the APs.
• We were having issues with live streaming at football games, so we re-tagged all the switches for SSID availability.
• We had multiple storms/ power outages
• Temporarily increase utilization (which has since gone back to normal, but problems haven't)

Thanks.

[Bossnine]

How much higher density?   And does the problems only affect the staff connection and not the student connection?

[SchoolITMan]

Building 1 has about 400 students and 85 Faculty/staff in about 43 rooms or avg 11.3 people per room,using about 32 APs, but only about 2-10 people per room using WIFI regularly.  The building footprint is 1 story and very spread out in long hallways due to multiple phase construction since over the last 75 years.

Building 2 has about 350 students and 55 faculty and staff in 26 rooms or avg 15.5 persons per room, using 24 APs, but they are ALL using WIFI.  The building is 10 years old and the footprint is square and 2 story.

Note that for both building these are just averages.  Students "migrate" from one end to the other for classes.  Clients per AP can exceed 50 or 60 and we have an AP in every other room.

Problems exist on both staff & student SSIDs, however, they affect them differently.  Student WIFI issues are rare, and kick everyone in 1 or 2 classrooms off at once.  Staff WIFI issues are more common, often repetitive throughout the day, and affect only a subset of the teachers.

Clear as mud?    :>\

[Bossnine]

Due to the current Covid situation are you doing more traffic intensive wifi applications, such as zooms, than you did before?

I'm also working in a school environment and our wifi is pretty rock solid, but I've seen issues in some locations where I had very high client density that previously had no issues but now due to google meets, etc on every chromebook experience some delays. 

50-60 clients on an AP, going through a wall, and doing some of this type of traffic might be an issue.  

[SchoolITMan]

Yes, zoom is our new top app.  Previously it was gmail / google classroom / google drive with a little youtube or Vimeo.  Now it is all of the above PLUS 40-45 faculty zooming all day.  And that doesn't even count students.

[Bossnine]

I'd definitely consider having an AP in every classroom with that kind of density and traffic requirements.  There may be more going on but I feel like your current load and environment would benefit from this..........if its possible that is.

[SchoolITMan · Accepted solution]

Just thought I'd post an update here as the issues SEEM to be fixed...

SOLUTION:

1. We rolled back a firmware update from 27.5.1 to 26.6.1. This fixed the authentication and speed issues.

2. We enabled QA on all VOIP and Video, and changed our Zoom DSCP tags to be standard so that QA would recognize them. (By default, Zoom is NOT set to 46/34 !)

3. We changed the radio profile settings from 20 power, 11 min bitrate on 2.4ghz and on 5ghz from 30 power, 24 min bitrate, and 40 chan width, to 22 power, 12 min bitrate, and 20 chan width. This fixed the signal overlap, AP hopping, and some of the interference.

4. We also had students disable hotspots on their iPhones, which they were doing because of the WIFI issues, but which was helping them connect, but was causing interference that was only making the issues worse.

RESULTS:

So far, since the changes, we have had no further complaints of WIFI or zoom dropping issues, despite the fact that our usage increased 2x since then.

[DeanPoirier]

Hi SchoolITMan, 

Curious where things are at with your setup... looking back at the solution that you posted, have you moved on from firmware 26.6.1? I'm facing similar issues, and looking at all aspects to try and resolve the regular user complaints.

[SchoolITMan]

Rolling back firmware fixed the issue.  Cisco acknowledged the firmware issue.  We are firmware locked and have not yet upgraded firmware since then, but plan to soon.

[cmr]

@DeanPoirier I think we have one AP that may be doing what is seen here, but it is an old MR32.  Our newer models (MR42 etc. through to 9166) run 29.5 and seem stable. What models are in your networks?

[SchoolITMan]

99% of our units are MR42.  We also have 1 MR56 and several outdoor model MR70s.

[DeanPoirier]

We do have a mix of MR32/33 and MR45/55 in place. The building that we receive the most feedback is running only MR32/33, so if that's still an issue on those older devices, that would make sense. Our main building is a mix of MR45/55 with 1 MR32. I have not taken note whether users are having the same issues in the main building, but I'm looking into that today. Thinking through whether I upgrade to 29.4.1 or roll back to 26.6.1... 

Thanks for your feedback!

[Knowledge_gainr]

Hi SchoolIITMan,

Can i get your contact no because we also facing same issue.