Wi-Fi constantly dropping in only one portion of a single physical network

[AaronKennedy]

Hi,

I am in charge of IT at a small K-12 school. Recently, some laptops on campus have begun to experience a perplexing issue ONLY when they are in one particular building. These laptops will automatically connect to an SSID broadcast by the MR42 units in that building, stay connected for 5-10 seconds, and then completely drop the wireless connection.  A few seconds later, the entire process repeats itself.  This will continue as long as the laptop is in that one building. If the owner takes the laptop to a different building, the laptop will connect to the same SSID broadcast by the MR34/42 units in that building, but will remain connected without any issues whatsoever. If the laptop owner subsequently returns to the affected building, they have about a 70% chance the issue will recur upon returning.

This behavior only happens with some laptops, and there does not seem to be any rationale for why it might be happening.  It has occurred with Acer (2015), HP (2016), HP (2018), and ASUS (2018) laptops, but only with some laptops from each affected model. Once a laptop has been 'afflicted', it will occur on all SSIDs broadcast by the MR units, regardless of which device on the network is operating as a DHCP server, or whether the client uses a DCHP lease, reservation, or statically assigned IP.

One guaranteed method of making this behavior happen on an otherwise perfectly functioning laptop is to attempt to connect to a UNC path (i.e. \\server\sharedprinter). As soon as a user tries to do this, their Wi-Fi connection will drop and the unusual behavior will begin.

There is no issue with wired devices in the affected building, and not all wireless devices exhibit the problem. Nothing on the dashboard indicates a problem with the Meraki hardware, and I have tested the cable connections from the gateway all the way back to each individual MR unit in that building.

I have tried forcing all devices to connect using only the 5GHz band, thinking there might be some kind of wacky RF interference in the 2.4Ghz band in that building, but it did not help.

It is not an issue of inadequate coverage or limited bandwidth. The building in question has twenty-four MR42 units covering 20,000 square feet (one MR unit per room in the building), and there are currently less than 100 wireless devices in the entire building.

Can anyone suggest what my next troubleshooting option might be?

[Rudi]

---

One guaranteed method of making this behavior happen on an otherwise perfectly functioning laptop is to attempt to connect to a UNC path (i.e. \\server\sharedprinter). As soon as a user tries to do this, their Wi-Fi connection will drop and the unusual behavior will begin.

---

This part stands out to me - how are your clients authenticating on the network? RADIUS? I would check your configuration for the APs in that building. 

Otherwise, do the laptops with problems use the same wireless chipset? It could be a driver issue. 

[PhilipDAth]

This sounds to me like someone or something is sending de-auth frames - especially being a school.

What does the Wireless event log say?

What does Air Marshall say?

Are you running 25.12 firmware?  Some of the older firmwares would occasionally mistake other Meraki AP's as being attackers.  A power cycle of the APs was needed to fix the issue.  I would try scheduling a reboot of all the APs when not in use.

[BlakeRichardson]

From the info you have provider the client device isn't the problem. I would be looking at the tagging on the switches, I suspect someone has changed a VLAn setting. When the devices connect are they getting an IP via DHCP?. I suspect they might not be.

[AaronKennedy]

Thanks Blake,

That was a possibility I had not considered.

However, when I examine the two switches in the affected building, both of them have L3 routing disabled, and neither are performing VLAN tagging or serving DHCP addresses. VLAN tagging is managed using group policy, and is based on either the SSID being used by the client device, or is manually set by myself when registering personal devices.

All of the affected devices are issued IP addresses by the Windows DHCP server running on our 2012R2 domain controller, and not by any Meraki hardware.  I have verified in the brief period of connectivity before the laptop disassociates that the correct IP address is being assigned to the client laptops by the DHCP server.

[BlakeRichardson]

@AaronKennedy Are there any logs on the Windows Server? Does it affect all SSID's or just a certain one?

[AaronKennedy]

The Windows server only handles DHCP and DNS for a single SSID.  The only devices that connect to that SSID are school-owned devices that need to communicate with our on-premises servers, printers, and data projectors.  All devices in that subnet have preset IP addresses using DHCP reservations, and IP addresses are not assigned to new devices that connect.

I just examined the DHCP logs on the server from yesterday and scanned for one of the affected laptops. It shows that laptop...

   1. Renewing its IP address

   2. DNS update request

   3. DNS update successful

That pattern repeats over and over every few seconds all day long. The only break in that pattern is when the teacher takes the laptop to another building.  Once there, the laptop connects and that is the last I see of it in the logs until it returns to the affected building.

[BlakeRichardson]

Can you run a packet capture on one of the switch ports for an affected access point. I have seen a very similar issue in the past but is was caused by bridging VLANS and DHCP requests travelling over two subnets that were bridged and meant there were duplicate ACK packets being sent. 

Is there anyone else your work with that may have changed something? has the firmware on your MR units updated recently? 

Something has to have changed to cause this. 

[AaronKennedy]

Hi Blake,

I will definitely run a packet capture the next time this I can confirm the issue is occurring on a laptop.  At this very moment, I am remotely copying a 5GB .iso file from the server to the teacher laptop that has been most affected by the constantly dropping Wi-Fi.  Of course, as soon as I tried this, everything started working normally on that laptop.

I will travel the building in a few minutes after students dismiss for the day and check teacher laptops for a 'victim' to act as a guinea pig.

[AaronKennedy]

Well.. just in case someone stumbles upon this thread in the future and wants to know if there was any resolution...

For about 10 days I had close to a dozen different laptops (all different makes, models & wi-fi chipset) that would spontaneously drop and reconnect their Wi-Fi connections all day long, regardless of which SSID they were connecting to. The network ecosystem at my school is a single physical network running entirely on Meraki hardware, yet this problem only occurred in one building on campus.

I did pursue all of the avenues of investigation that were suggested by those who responded to this thread. In the end, I made no changes to the network.  Last Thursday afternoon, the problem simply vanished.  I could not explain what caused it, and I cannot explain what made it go away.

There are still some other gremlins inhabiting the network in that building, but that is a topic for a different thread.

[AaronKennedy · Accepted solution]

Sorry to resurrect a dead thread, but after two years I finally found what seems to be both the source and the solution to this problem...

ISSUE: Some (random) laptops were constantly dropping their Wi-Fi connections while in one of the buildings on campus, but were perfectly fine when in other buildings (entire campus on the same 100% Meraki network).

WORKAROUND: Most laptops resolved the problem spontaneously after about 2 weeks. For those that did not, using a USB thumb wireless adapter seemed to solve the problem.

SOLUTION: It turns out the SSID Wi-Fi profile that was using the internal NIC on the affected laptops was set to 'current user' instead of 'all users'. As soon as I used netsh wlan commands to set the wireless profile to 'all users', the machines with the persistent problem were fixed. 

[Marc_Abaya]

Aaron, great post. I'm having similar problems, but my Wireless Profile was pushed by Group Policy. When you enter netsh wlan show profile, it will show under "Group Policy profiles (read-only)" and not "All User profile".

Now I'm curious how you can move them to "All User profile".

Our workaround is the same as yours, we provide them USB WIFI adapter which fixes the problem.

[AaronKennedy]

Although all of my staff laptops are members of an Active Directory domain and are managed by Group Policy, I do not manage wireless profiles on those devices using a group policy. The staff member who uses each laptop actually signs into the device using a local account, not a domain account. This means I can use a local administrator account on the affected machines to delete the faulty profile and then import an "all user" profile for the same SSID.

School laptops that are used by students do have wi-fi managed through group policy (primarily to prevent them from connecting to cell phone hotspots as a way to bypass the school firewall). However, I have never experienced this issue on those laptops.

After doing some experimenting over the last few weeks, I think the issue can be avoided if I join the device to wi-fi at the right time during setup.

     1. Install a (non-domain) image onto the device over ethernet using MDT.

     2. Connect to wi-fi using the local administrator account.

     3. Join the laptop to the domain (over wi-fi).

     4. Create the local account for the end-user.

If the device is domain-joined with MDT as part of sysprep or domain-joined over ethernet after setup has finished and THEN connected to wi-fi, the resulting wireless profile will be "per user", but will not exhibit any issues.

If the local user account is created and that account is used to join wi-fi, then not only will the wireless profile be "per user", but the wi-fi connection will drop whenever a running task owned by 'SYSTEM', 'LOCAL SERVICE', or 'NETWORK SERVICE' attempts to access something over the network.

[AaronKennedy]

Hi Philip,

I agree that the behavior is reminiscent of a prankster sending out poison packets, but there are a couple of arguments against that...

     1. This building houses our primary grades (K-3), and those students do not have personal mobile devices or laptops at school.

     2. If someone was spoofing de-auth frames, then there would be more affected teachers/devices.  As it is right now, only a small number of teachers in the building have affected laptops, and the behavior persists even in the evening after all students have gone home.

When examining the wireless event log for the affected computers, there is a repeated pattern of

   * 802.11 association

   * WPA authentication

   * 802.11 disassociation

This behavior repeats from a maximum of every 15 seconds for some affected laptops, to a minimum of every 3-4 minutes for others (but not at all for most laptops)

Air Marshal only detects the three printers on campus that also broadcast a Wi-Fi Direct signal to enable printing from tablets.  Nothing else unusual shows up there (aside from the hundreds of Wi-Fi broadcasting vehicles that drive past the school each day.

All 54 access points on campus are running 25.12 firmware.  I actually rebooted all APs on campus (not just the ones in the affected building) to no avail.

[AaronKennedy]

Thanks Rudi,

I added that bit about UNC paths because it seemed a very unusual trigger for this behavior.  To answer your questions...

1. The SSID that these devices connect to is only used by our domain workstations, and they authenticate using Active Directory using the built-in Meraki captive portal.  On that VLAN, DHCP will only issue an IP address to pre-registered devices that have an IP reservation.  There are no RADIUS servers on our network.

2. All the access points on campus have an identical configuration.  That is why the behavior in this one building is perplexing.  It cannot be replicated in any other location on campus (even though all buildings are part of the same physical network).

3. I investigated the possibility that the wireless chipset on the laptops could be the issue. However, of the four different models of laptop that have been affected so far, and although all four use RealTek chipsets, there are three different chipset models between them.