Ethernet port carrier change on certain upstream (WAN) switches connected to MX appliance

[ely105]

Hi, 

I've noticed that on some upstream switches to some of my MX appliances(MX75) on the WAN connection I get occasional "Ethernet port carrier change" events and notice that there is a definite disconnect and reconnect.  Usually its only for 5-30 seconds and can go hours without one and then have many.  I know many posts say to check cabling but with the same exact cabling i can swap out the switch and it completely disappears.  It was a little ironic when i first noticed it was when I was using some 8 port Cisco SBS switches.  I have found that the Netgear GS108T seem to be most reliable and doesn't exhibit this problem.  I recently saw the problem with a small 8 port HP switch as well.  Since it doesn't appear to be cabling related I wanted to find out if anyone else has seen this as well?  I can open a support ticket and have them investigate as to maybe more details if needed.  Just thought i'd ask here first.  Thanks

[OVERKILL · Accepted solution]

There's a known issue with Energy Efficient Ethernet being enabled on a non-Meraki switch that will cause flapping. 

[ely105]

Great suggestion. Let me take a look at that. 

[PhilipDAth]

This could also be the MX crashing.  Make sure you are on a current stable firmware release.

You can double-check by opening a support ticket and saying you are wondering if the MX has crashed and asking for the uptime.

Are you operating the MX inside of the sizing guide (especially with regard to the number of users)?

https://meraki.cisco.com/product-collateral/mx-sizing-guide/?file 

[ely105]

Good suggestion!  But the load on these MX’s is next to nothing many times. And I can clearly see the behavior disappear when replacing upstream switch. But good to know. There is no other way to detect a crash/reboot without opening a ticket?

[OVERKILL]

When I had an MX84 hitting 100% and flapping the interfaces, it was all the interfaces that were active. If you are only seeing the WAN interface flap, I'd suspect it isn't a crash, as all should flap under a crash as well. 

[mwiater]

We too saw this behavior on MX 85's with Cisco SG110 switches, and possible Netgear 105s (we don't have a record of what we swapped in for our test), on the wan side.

My ticket is still awaiting input from engineering, but we had to remove our MX-85's from production as it was impacting voice.

If anyone has more information, or a recommended switch, would appreciate input.

[mwiater]

I should have mentioned, we're confident our problem is unrelated to crashing nor load issues.  And we don't see the problems on MX-84's that we swapped in place of the 85's.

[OVERKILL]

I just checked the SG110 and it supports EEE, so that may be your issue:

[OVERKILL]

Just checked the Netgear GS105 and it also supports EEE:

[OVERKILL]

Do you have access to an older Cisco switch that for sure doesn't have EEE? I'm using a stack of 2960S's at a client and they definitely don't have it and they are rock solid. 

[mwiater]

we're looking around but so far, no, we don't. Good to know there's a model that does not exhibit these problems. Very painful.

[OVERKILL]

No problem. As I posted above, both of those switches you are using definitely support 802.3az (EEE), so that's likely the problem. Anything from the Cisco 2960-S family doesn't:

https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-2960-s-series-switches/data_shee...

If you scroll down to standards, you'll see that 802.3az is not listed. 

Lots of them out there, I have a pile of them here. 

[ely105]

I can confirm that the Netgear GS108Tv2/v3 does not have EEE(Energy Efficient Ethernet) enabled by default and has not given us flapping as an upstream switch to a MX appliance.

I saw the issue on the Cisco CBS-250 8 Port switch (similar size/features as GS108T), which has EEE enabled by default.  I have since disabled it and it appears the flapping problem is gone now.  It just means they have to be configured to disable EEE before using upstream of a MX.

[JonP]

We have the same problem, only slightly reversed. We saw the MX drop WAN1 at random - only for a second or two and then come back up. We were directly connected to the ISP router and they were saying it is the MX dropping, whilst Meraki were suggesting it is the ISP router. We installed an HPE Aruba 2920 in between to log the port changes and we can see that the MX is indeed dropping the WAN1 interface, whilst the ISP router stays up.

We logged these findings with Meraki and they are replacing the MX as they believe it is a faulty port. 

[OVERKILL]

The Aruba 2920 family also supports 802.3az, which is what is known to cause the flapping. Your ISP router may as well. See if you can disable it on the port that the MX is connected to. 

[JonP]

Apologies, it's a 2530, and I don't think EEE is enabled? 

HP-2530-48G(config)# sh savepower led

 LED Save Power Information

 Configuration Status : Disabled

HP-2530-48G(config)# sh savepower port-low-pwr

 Port Save Power Information

 Configuration Status : Disabled

HP-2530-48G(config)# sh energy-efficient-ethernet

  Port  | EEE Config Current Status txWake(us)

  ----- + ---------- -------------- ----------

  1     | Disabled   Inactive       -

  2     | Disabled   Inactive       -

[OVERKILL]

Yes, that does indeed look disabled. See if the problem goes away with the new unit, if it doesn't, I'd investigate this angle more deeply. The flapping I experienced on an MX84 was all active interfaces and was related to AnyConnect getting hammered and not related to the device. 

[thomasthomsen]

Just had this EXACT problem on a MX85 (both WAN ports).

Disabled EEE on the uplink switch, port is now stable.

Do anyone know if there is a bug on this ? (its not anywhere in the release notes).