Switch Device Health MS425-C9300 Meraki Managed

[TNAComputers]

First off, I realize this feature is early access/beta, but I have some concerns and I wanted to see if anyone else seen this. I have a MS425 switch that I enabled this and its showing an alarming number of drops. This switch is half populated and maybe early morning some interfaces get maxed out for high bandwidth operations. However throughout the day I will see 10k plus drops just randomly (not during high utilization times).  

You can see for the last week 60k packets were dropped over 4 hours and that's not the highest. Millions of packets are getting dropped according to this tool per month. Outside of this tool, there is no way to tell. The event log shows no errors and there isn't a way to correlate this information. I opened up a ticket with Meraki, and they said "Reaching out to our internal team to confirm and they stated the traffic being dropped is not client traffic. It is traffic copied to the management CPU for inspection for functions such as client tracking". I believe this would still cause issues is millions of packets are dropping per month esp with CPU hits eating up resources. 

I have issues with accurate client tracking as well for some time. It was set to MAC address for client tracking as I used to have a C9300 behind the MS425 as a "access switch" Its since been converted to Meraki Managed so it was changed to Unique client identifier tracking, since that's whats recommended. 

Has anyone else seen issues like this with high packet drops for those that have this turned on? I don't have another device in another org/network to look at like this. 

[RWelch]

What firmware is your MS425 running?  

[TNAComputers]

17.2.2 on the 425, and CS 17.2.2 on the 9300

[RWelch]

MS 17.2.2 has been steady (reliable) for my networks - that is good news!  I'd stay with firmware MS 17.2.2. 

[RWelch]

It would also be worth checking to see what your MS425 MTU size is at since there is hardware limitation specific to MS425 series switches - I am not sure if this might be a contributing factor in your drops.

Switch Settings 

• The maximum MTU on MS425 is 9416 bytes (i.e.: 9394 + 22 = 9416). Therefore, enter MTU size 9394.

[TNAComputers]

Its just the default 9198. Not sure where it pulled that number, but I have never checked it before so im assuming that's what it was set to when I added to the network. 

[RWelch]

It's likely set at that due to the limitations of the C9300 series.

Are you using SFP+ or DAC cables?

Is the module seated secure into the port?  I have had instances that the module pulled out without me releasing or latch/release tab.

[TNAComputers]

Im using DAC cables. Interesting story is that I was having loss from the MX to the 425 last year. I had Cisco branded SFPs and fiber patch cables.  Meraki suggested that I swap those for some specific DAC cables, and it did clear up the issue (all SFPs were replaced with DAC cables). When I opened up this case, they were questioning these DAC cables, so I had to reference the previous case where they suggested them 🙂 

It is odd though that it shows OEM and gives basic info for those SFPs in the dashboard. TAC said they expected to see more info almost like they are not 100% supported...

I will double check to make sure all of them are seated correctly. I was able to look at someone else's dashboard with a MS225 and it does show 10k plus drops as well, so it might be a early access thing etc, or they are really valid, and they have only Ethernet connections.

[RWelch]

For what it's worth - I had temporarily used DAC cables in my early deployment but elected to go with 1M AOC instead - BIG difference (so what if it's 3rd party) but I no longer have any drops.  

And the drops I had wasn't on the scale you mention - it's just a lab/test (sometimes demo) network with not much data (throughput).

[RaphaelL]

What types are these packets ?

I have 1513 switches on a total of 3459 that are showing > 0 drops. I have 0 problems on all of these sites.

For example some switches are reporting a large amount of drops, these are STP drops because BPDU guard is triggered. 

[TNAComputers]

They are mostly in the other category. I'm looking into multicast issues with the 9300 per @RWelch to see if the drop issue is causing this to send multicast traffic to the 9300 then to the 425 and thats whats causing it. Im seeing a bunch of IGMP/multicast traffic from the 9300 to the 425

[PhilipDAth]

In the distant past, I have seen this happen due to mass spanning tree calculations, as the switches had every port configured as a trunk port.  This specific issue typically occurs when a large number of people are coming or leaving the network location.

Every time a trunk port goes up and down, anywhere in the entire network, on any switch, it triggers a spanning tree re-calculation.

To avoid this, it is essential to ensure that switch ports connected to end-user devices are marked as access ports.  Spanning tree ignores access ports when they go up and down.

[TNAComputers]

Thanks for the suggestion @PhilipDAth . I am pruning all vlans that are unneeded on trunk ports, and all end user ports are access. There are a few trunk ports, but they are to APs and to the MS425. I checked the event log on the 425 and no port bounces since the last time I upgraded the firmware. On the access side, no trunk ports have bounced. 

I double checked the device health, and no STP drops. Just other. 

[TNAComputers]

As an update, Meraki has looked into this issue, and said this is normal. I don't see how millions of packet drops over the course of week/month is a normal thing, but I have no way to prove otherwise. 

Coming from a Cisco Catalyst background, 100k+ packet drops over a day/week would be a high cause for concern, or at least worth looking into to see what is causing it. The main question is does this have an effect on performance? No, not that I can prove. Other than me seeing a number on a page, I cannot gather any additional information on what is happening. I have checked all of the Meraki docs in regard to best practices on MS, and those have been applied. It does appear to be better, but still high at times. Since its random, I cant get a packet capture to see from the switch, other than normal traffic. 

Iperf tests between ports, and L3 boundaries appear fine (on the same 425 port to port) and on the uplinks from the 425 to the 9300. No errors on the physical ports on either switch (for uplinks or trunked ports). Nothing in the event logs for either device. Errors are not incrementing when performing the tests, so whatever is going on is specific to the management traffic. 

[TNAComputers]

Hi Everyone **I posted this in the MX side, but I think its related to the drops on the 425**

I think the QoS rule was hitting and for whatever reason was the cause of the high drops. Im still seeing drops, but nowhere near the amount since the QoS rule was removed. As mentioned this was in a different vlan not any from the onces I was testing on. IPERF speeds were good between hosts and switch to switch. Up to the MX was the issue from the switch. Ill keep everyone posted on this, but I think this was all related. 

I thought I would post back letting you know this is resolved. Thanks for everyone who looked at this issue and took the time to respond.

I spent alot of time running iperf tests between switches, hosts etc, and was seeing the correct speed between hosts. Even on the 425 I was getting the full 10gbps speeds between hosts. I couldn't directly test from the 425 to MX. However, I plugged directly into the MX and got full speeds, but couldn't behind the 425. I went through every setting for MX,MS,MR,MV etc and I mean EVERYTHING. I found a QoS rule for Vlan 800 (not my vlan I was testing from) changing to EF. All shaping was off, all other QoS was off. I removed that rule on the 425 and everything went back to normal. I have no idea why this was happening. I gave this info to TAC and if they find RCA I will let everyone know.