Ethernet port carrier change on certain upstream (WAN) switches connected to MX appliance

SOLVED
ely105
Here to help

Ethernet port carrier change on certain upstream (WAN) switches connected to MX appliance

Hi, 

I've noticed that on some upstream switches to some of my MX appliances(MX75) on the WAN connection I get occasional "Ethernet port carrier change" events and notice that there is a definite disconnect and reconnect.  Usually its only for 5-30 seconds and can go hours without one and then have many.  I know many posts say to check cabling but with the same exact cabling i can swap out the switch and it completely disappears.  It was a little ironic when i first noticed it was when I was using some 8 port Cisco SBS switches.  I have found that the Netgear GS108T seem to be most reliable and doesn't exhibit this problem.  I recently saw the problem with a small 8 port HP switch as well.  Since it doesn't appear to be cabling related I wanted to find out if anyone else has seen this as well?  I can open a support ticket and have them investigate as to maybe more details if needed.  Just thought i'd ask here first.  Thanks

1 ACCEPTED SOLUTION
OVERKILL
Building a reputation

There's a known issue with Energy Efficient Ethernet being enabled on a non-Meraki switch that will cause flapping. 

View solution in original post

18 REPLIES 18
OVERKILL
Building a reputation

There's a known issue with Energy Efficient Ethernet being enabled on a non-Meraki switch that will cause flapping. 

Great suggestion. Let me take a look at that. 

PhilipDAth
Kind of a big deal

This could also be the MX crashing.  Make sure you are on a current stable firmware release.

You can double-check by opening a support ticket and saying you are wondering if the MX has crashed and asking for the uptime.

 

Are you operating the MX inside of the sizing guide (especially with regard to the number of users)?

https://meraki.cisco.com/product-collateral/mx-sizing-guide/?file 

Good suggestion!  But the load on these MX’s is next to nothing many times. And I can clearly see the behavior disappear when replacing upstream switch. But good to know. There is no other way to detect a crash/reboot without opening a ticket?

OVERKILL
Building a reputation

When I had an MX84 hitting 100% and flapping the interfaces, it was all the interfaces that were active. If you are only seeing the WAN interface flap, I'd suspect it isn't a crash, as all should flap under a crash as well. 

mwiater
Here to help

We too saw this behavior on MX 85's with Cisco SG110 switches, and possible Netgear 105s (we don't have a record of what we swapped in for our test), on the wan side.

 

My ticket is still awaiting input from engineering, but we had to remove our MX-85's from production as it was impacting voice.

 

If anyone has more information, or a recommended switch, would appreciate input.

I should have mentioned, we're confident our problem is unrelated to crashing nor load issues.  And we don't see the problems on MX-84's that we swapped in place of the 85's.

OVERKILL
Building a reputation

I just checked the SG110 and it supports EEE, so that may be your issue:

 

Screen Shot 2022-04-05 at 3.13.29 PM.png

OVERKILL
Building a reputation

Just checked the Netgear GS105 and it also supports EEE:

 

Screen Shot 2022-04-05 at 3.16.27 PM.png

OVERKILL
Building a reputation

Do you have access to an older Cisco switch that for sure doesn't have EEE? I'm using a stack of 2960S's at a client and they definitely don't have it and they are rock solid. 

we're looking around but so far, no, we don't. Good to know there's a model that does not exhibit these problems. Very painful.

OVERKILL
Building a reputation

No problem. As I posted above, both of those switches you are using definitely support 802.3az (EEE), so that's likely the problem. Anything from the Cisco 2960-S family doesn't:

https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-2960-s-series-switches/data_shee...

 

If you scroll down to standards, you'll see that 802.3az is not listed. 

 

Lots of them out there, I have a pile of them here. 

ely105
Here to help

I can confirm that the Netgear GS108Tv2/v3 does not have EEE(Energy Efficient Ethernet) enabled by default and has not given us flapping as an upstream switch to a MX appliance.

 

I saw the issue on the Cisco CBS-250 8 Port switch (similar size/features as GS108T), which has EEE enabled by default.  I have since disabled it and it appears the flapping problem is gone now.  It just means they have to be configured to disable EEE before using upstream of a MX.

 

 

JonP
Getting noticed

We have the same problem, only slightly reversed. We saw the MX drop WAN1 at random - only for a second or two and then come back up. We were directly connected to the ISP router and they were saying it is the MX dropping, whilst Meraki were suggesting it is the ISP router. We installed an HPE Aruba 2920 in between to log the port changes and we can see that the MX is indeed dropping the WAN1 interface, whilst the ISP router stays up.

 

We logged these findings with Meraki and they are replacing the MX as they believe it is a faulty port. 

OVERKILL
Building a reputation

The Aruba 2920 family also supports 802.3az, which is what is known to cause the flapping. Your ISP router may as well. See if you can disable it on the port that the MX is connected to. 

JonP
Getting noticed

Apologies, it's a 2530, and I don't think EEE is enabled? 

 

HP-2530-48G(config)# sh savepower led
 
 LED Save Power Information
 
 Configuration Status : Disabled
 
HP-2530-48G(config)# sh savepower port-low-pwr
 
 Port Save Power Information
 
 Configuration Status : Disabled
 
HP-2530-48G(config)# sh energy-efficient-ethernet
 
  Port  | EEE Config Current Status txWake(us)
  ----- + ---------- -------------- ----------
  1     | Disabled   Inactive       -
  2     | Disabled   Inactive       -

 

 

OVERKILL
Building a reputation

Yes, that does indeed look disabled. See if the problem goes away with the new unit, if it doesn't, I'd investigate this angle more deeply. The flapping I experienced on an MX84 was all active interfaces and was related to AnyConnect getting hammered and not related to the device. 

thomasthomsen
A model citizen

Just had this EXACT problem on a MX85 (both WAN ports).

Disabled EEE on the uplink switch, port is now stable.

Do anyone know if there is a bug on this ? (its not anywhere in the release notes).

Get notified when there are additional replies to this discussion.
Welcome to the Meraki Community!
To start contributing, simply sign in with your Cisco account. If you don't yet have a Cisco account, you can sign up.
Labels