MR30H inconsistently forwarding DHCP, not allowing communications between wired and wireless clients

[FlyingPan1994]

Hello all,

I am coming here after raising several support cases about two major issues, and not really getting anywhere closer to a solution, so I am hoping someone here had similar issues in the past, and was able to resolve them somehow.

We have about 50 MR30H access points spread throughout almost as many remote sites. We first started noticing issues around February of 2020. In a nutshell, the MR30H APs will report they are connected to the Meraki Cloud, and to our MX-100 running as a VPN concentrator at the main office. The MR30H APs have wireless clients connecting to them, and also have wired printers and desktops connected to them. In most cases, these remote sites have separate, company owned and controlled internet connections used by the APs. In some cases, we need to work with the remote site's IT department to set up a port for the APs that basically just dumps straight out onto the Internet for us.

The first major issue is with DHCP. I have our DHCP server set up at the main office, where the MX-100 is. Sometimes, but not always, the MR30H APs will report that clients are on a particular VLAN, but they never get an address from the DHCP server. They will fail, then pull one from the AP itself in the 10.x.x.x range, which isn't right. Sometimes the same AP will "allow" the DHCP lease to come through, sometimes it wont. It seems to be more of an issue on wired clients than wireless ones. Then the next day, that AP is fine, but a different one will start acting up the same way. Support claims my DHCP server is not responding properly, despite it working fine for every other client in every way, and its basic settings not being changed for years. 

The other major issue is communication between wired and wireless clients. Specifically, printers. We have wired printers at many of our remote sites. The wired clients can always connect to the printers just fine, but wireless clients on that AP, and also clients at the main office, cannot ping or connect to that printer *until* I ping it using the Ping tool on the AP itself. I get "destination host unreachable" until I ping from the AP, then suddenly I get responses on wireless and remote clients. The wired desktops never have an issue, and the printer always responds for them. If I stop pinging for 60 seconds, I go back to "destination host unreachable" until I ping from the AP again. The wired desktops never drop a single packet to the printers during this time. Many support cases and hours of packet captures with support later, and still no answer. This is happening on multiple MR30H APs at multiple sites, with multiple printer makes and models.

I have been told:

To put the MR30H APs into a separate Meraki network for each site

To create a VLAN and Subnet for each site, and put both wired and wireless clients on that VLAN and Subnet

To make a separate DHCP scope for each site

To avoid using "wired only" SSIDs and instead use Enabled with Hide SSID for any AP port profiles using those SSIDs

Update firmware

That MR30H APs are finicky and weird (came straight from support)

I have done all of these things, but nothing has helped. Has anyone run into issues like this? I'm ready to buy 50 FortiGates and set up proper site-to-site VPNs if I can't get this working.

Thanks!

[PhilipDAth]

I think a packet capture is going to be needed to get some straight answers on this one.

Next time you get a case where DHCP is not working, do a DHCP capture on the MR30H.  Does it see a DHCP request come from the client?  If so, do you see a response?

[FlyingPan1994]

Hello PhilipDAth,

Thanks for the reply. We did get a packet capture when one of the APs was having this DHCP issue. Here is what we found on a port connected to a device that was not getting an IP from our server:

Format:

[Time]: [Source IP] > [Destination IP] - [Info]

0.000 sec: 0.0.0.0 > 255.255.255.255 - DHCP Request

0.1578 sec: 0.0.0.0 > 255.255.255.255 - DHCP Discover

0.1579 sec: 192.168.160.1 (Default gateway of Subnet) > 192.168.160.87 - DHCP Offer

0.1582 sec: 192.168.6.224 (One of our DHCP Servers) > 255.255.255.255 - DHCP Offer

4.727 sec: 0.0.0.0 > 255.255.255.255 - DHCP Request

4.728 sec: 192.168.160.1 > 255.255.255.255 - DHCP NAK

4.730 sec: 192.168.6.224 > 255.255.255.255 - DHCP NAK

7.799 sec: 192.168.6.224 > 255.255.255.255 - DHCP Offer

10.630 sec: 0.0.0.0 > 255.255.255.255 - DHCP Request

10.959 sec: 0.0.0.0 > 255.255.255.255 - DHCP Request

10.960 sec: 192.168.160.1 > 255.255.255.255 - DHCP NAK

10.970 sec: 192.168.6.224 > 255.255.255.255 - DHCP NAK

14.037 sec: 192.168.6.224 > 255.255.255.255 - DHCP Offer

14.076 sec: 0.0.0.0 > 255.255.255.255 - DHCP Request

14.078 sec: 192.168.160.1 > 255.255.255.255 - DHCP NAK

14.080 sec: 192.168.6.224 > 255.255.255.255 - DHCP NAK

This pattern continued for as long as the capture was taken. This manifests on the Clients tab of the AP as the clients being on the correct VLAN, but after a while they grab a 10.x.x.x IP addresses from the AP itself. I have also tried enabling the Mandatory DHCP option, but this does not help when we are having this issue.

Other MR30H APs in other locations do not have this issue, and they are connecting to the same MX-100, at the same location, with the same DHCP servers. Sometimes an AP will have this issue one day, then not the next. We started to think it was an ISP issue, but we can have APs working and not working, which use the same ISP in different locations. In this particular case, we even replaced the MR30H, which fixed the issue for a couple of weeks, but then it returned.

[PhilipDAth]

It shows the client is being offered an address - but the client never accepts it.

>0.1579 sec: 192.168.160.1 (Default gateway of Subnet) > 192.168.160.87 - DHCP Offer

>0.1582 sec: 192.168.6.224 (One of our DHCP Servers) > 255.255.255.255 - DHCP Offer

Is 192.168.160.1 configured to forward DHCP requests to 192.168.6.224?

[FlyingPan1994]

Yes, it sure is. Every default gateway interface we have is on our Meraki core switch, and they are all configured to forward DHCP requests to our on-site DHCP servers:

Internet connection is connected to our Firewall

Firewall is connected to our Core Switch

Core Switch has all default gateway interfaces and VLANs for all of our subnets, and is routing between them.

Core Switch is connected to MX-100.

Core Switch is connected to DHCP Servers.

Remote MR30H APs tunnel back to MX-100.

MX-100 places MR30H traffic on particular VLANs depending on which remote site they are in.

We'll have this IP/DHCP issue with phones, laptops, and desktops, but generally it's with wireless devices. It's multiple types of devices in multiple locations. 

Also, thanks for actually looking at the capture. Support told us it was evidence that our DHCP server wasn't working, which clearly it is. It's all been quite frustrating.

[FlyingPan1994]

Had another support call on Tuesday. This is the third call where they have taken packet captures, then told me the engineering/dev team will have to analyze. This was for the communication issue between wired printers and wireless devices connected to the MR30H.

Wired devices can always ping the printers perfectly, with no dropped packets. Specifically, I test a wired desktop PC pinging a printer. When a wireless device attempts to ping that same printer, the first two packets time out, then I get recurring "Destination Host Unreachable", even though all of these devices are using the same VLAN and subnet. Sometimes, after about a minute or so, the printer will then suddenly be able to reply. This happens even with a continuous successful ping going from one of the wired desktops to the printer at the same time. I can force the printer to reply to wireless clients immediately if I ping it from the AP using the Ping tool from the console. Then, if I wait 60 seconds, I can no longer ping from wireless devices, even with the continuous one still going from the wired desktop.

It almost seems like some kind of "tunnel" closes between the wired printers and wireless devices if there is no communication after about 60 seconds or so. Support has remarked that what is happening is very weird, and unexpected. I'm having this exact same issue with multiple printers from different vendors in multiple locations. Our workaround is to have a continuous ping going from our print server at the main office. For some reason, this holds that "tunnel" open between the wireless and wired printers at the remote sites, even though this server is at the main office and is on a totally different VLAN.