Ethernet PAUSE frame count increasing on MR18-attached port

[RJordan-CCS]

The switch is a Cisco Small Business SG350-48x.  We have four Meraki APs connected to individual ports on it. The site has been complaining about periodic slowdowns on the network, though not restricted to clients connected to the MR18; we haven't managed to ID a specific problem so far but found this issue while analyzing.

On one of the ports with an MR18 as the only connected device, we are getting a continuously escalating PAUSE frame counter. There is no indication of overutilization, no loops, drops, STP errors etc on the switch and on its initial check the Meraki tech said they saw no indication the AP was overutilized.  The PAUSE counter increases continuously whether or not clients are connected to the MR18 and we are sure there are no double-connected devices (wireless and wired), though since they are on different subnets and VLANs that shouldn't be an issue.

We've also run a wire test with our Fluke gear and its good.  Have not had time to do an AP swap.

We set up a SPAN monitor port and have run captures (we do have an open ticket with Meraki but they haven't gotten back to us recently so asking here).   The PAUSE frames are seen between 8 and ~60 times per 10 second interval, and continuously, not just during periods of perceived slowdown (so possibly unrelated).  They come in pairs, with one giving pause time of 65535 quanta and the next a pause time of 0 quanta, repeated endlessly.

The MAC address being reported as the source for the pause frames doesn't show up anywhere else.  It isn't the MR18's MAC, nor any of the attached clients, doesn't show up in ARP tables anywhere on the client network, including the switch we're recording the pause frames on, and also doesn't show up in any MAC vendor database that I've been able to find

c5:20:75:0a:18:00

3 15:19:46.907701 c5:20:7f:0a:18:00 Spanning-tree-(for-bridges)_01 MAC CTRL 60 Pause: pause_time: 65535 quanta

4 15:19:46.907701 c5:20:7f:0a:18:00 Spanning-tree-(for-bridges)_01 MAC CTRL 60 Pause: pause_time: 0 quanta

I believe these are BPDU frames (Bridge Protocol Data Unit), but the MAC doesn't correlate to the multicast MAC I read about online, and I haven't been able to figure out why we are seeing these on the one port with an AP connected and no others.

Any thoughts?  I may be imposing a bit on Meraki since its not a Meraki switch, but the AP is Meraki, and as the only device connected to that port, it presumably has to be the source of the PAUSE frames.

Thanks
Rich

[ww]

did you try disable flow control on the switchport

[PhilipDAth]

I'm not very confident that a pause frame would cause your issue, but I would go with @ww solution of disabling pause frame support on the switch port.

A Gigabit port could easily out pace an MR18, leading to buffer exhaustion.  Asking the switch to slow down the traffic flow seems a very reasonable thing to me.

I would try disabling the 2.4Ghz band.  If you are using that it is far more likely to cause intermittent performance issues.

Also, the MR18 went "end of sale" quite some time ago.  You could consider upgrading to a more recent model.  Note that they use the same licence, so as long as you removed the old AP, you don't need to buy a new licence - just the new AP itself.

I would try and go for a minimum of an MR33 if you are having issues.  That is because this is the first model to have a dedicated scanning radio for mapping out RF problems (and security issues).

If you want to get right up to date I would take a look at the MR45, as it is WiFi6 capable.  Note you need 802.3at power, so if your switch can not deliver this you will need a PoE injector or a power transformer.

[RJordan-CCS]

Thanks all for replying.

We haven't been able to try switching to another port; no tech onsite until later this week.  We can try.  Also can try the flow control disabled, but we have three other APs (two MR18 and two, I think since I'm away right now MR42s).  All of them are plugged into identically configured switch ports on this same switch. 

The other MR18 connected port has seen a few pause packets (the count is in the low thousands since last July; the problem port is over 70,000,000 since July.

Next time the tech is there I'll have him try another port for the AP.  If the problem follows the move (which i expect) we can modify flow control remotely.

The MR18 is (apparently) pumping out pause packets even when no clients are attached. 

[RJordan-CCS]

OK, AP1 and AP2 are MR32s, AP3 and AP4 are MR18s.

AP4 is generating all the pause frames on this port; AP3 counters are four orders of magnitude smaller over the same period of time.

We moved AP4 to an unused switch port after cloning the existing port config (for VLANs, etc). So far 3000 pause frames received in about 30 minutes, with (currently) three clients attached; a printer, an iphone, and a kiosk desktop. In total over the last 30 minutes only about 100MB has been transferred. so it doesn't appear to be too busy.

Ran another capture via the SPAN port and it looks the same. The reported source MAC didn't change; I assume its a 'generated' MAC since I can find no sign of it anywhere else, nor documented online. Reviewing details the 'destination' MAC is 01:80:c2:00:00:01 which is a reserved MAC for pause frames (this is new to me; I've not delved into this level before).

The port flow control is already disabled (and was on the original port). I enabled it temporarily and the port got flooded with pause packets, nearly 10000 of them in a few minutes. Then it slowed back down to the previous rate.

I switched it back to disabling flow control.

No events logged, and the one client I tested (I was managing the kiosk which connects via AP4) saw no indications of trouble.

Maybe there is something about this particular MR18 that is wonky. I guess we'll wait to see once tech support gets back to us; I'm updating that case now.

Thanks

Rich

[kYutobi]

Have you tried using a different port? Does it follow the MR unit?