MAC is flapping between two MX's

[Ramakrishnan]

Dear All,

I have two MX 85 for warmspare connected to cisco switch and i am seeing lots of MAC flaps on my switch where meraki MX 85 are connected as warm spare.

Feb 13 15:25:22.066: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
Feb 13 15:25:23.066: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
Feb 13 15:25:36.424: %SW_MATM-4-MACFLAP_NOTIF: Host cc03.d985.3bb4 in vlan 10 is flapping between port Gi1/0/2 and port Gi1/0/1
Feb 13 15:31:18.225: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
Feb 13 15:31:19.225: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to down
Feb 13 15:31:21.902: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
Feb 13 15:31:22.905: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
Feb 13 15:31:36.173: %SW_MATM-4-MACFLAP_NOTIF: Host cc03.d985.3bb4 in vlan 10 is flapping between port Gi1/0/1 and port Gi1/0/2
Feb 13 15:31:38.222: %SW_MATM-4-MACFLAP_NOTIF: Host cc03.d985.3bb4 in vlan 10 is flapping between port Gi1/0/2 and port Gi1/0/1
Feb 13 15:32:29.343: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
Feb 13 15:32:30.346: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to down
Feb 13 15:32:33.027: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
Feb 13 15:32:34.030: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
Feb 13 15:32:47.183: %SW_MATM-4-MACFLAP_NOTIF: Host cc03.d985.3bb4 in vlan 10 is flapping between port Gi1/0/1 and port Gi1/0/2
Feb 13 15:32:53.233: %SW_MATM-4-MACFLAP_NOTIF: Host cc03.d985.3bb4 in vlan 10 is flapping between port Gi1/0/2 and port Gi1/0/1
Feb 13 16:52:33.663: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
Feb 13 16:52:34.666: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to down
Feb 13 16:52:42.866: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
Feb 13 16:52:43.869: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
Feb 13 16:58:38.945: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
Feb 13 16:58:39.948: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to down
Feb 13 16:59:30.189: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
Feb 13 16:59:31.192: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
Feb 13 17:08:42.624: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/2, changed state to down
Feb 13 17:08:43.627: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Feb 13 17:09:01.904: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Feb 13 17:09:02.907: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/2, changed state to up
Feb 13 19:06:20.187: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
Feb 13 19:06:21.186: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to down
Feb 13 19:06:23.867: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
Feb 13 19:06:24.867: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
Feb 13 20:41:28.106: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
Feb 13 20:41:29.106: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to down
Feb 13 20:41:31.745: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
Feb 13 20:41:32.745: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
Feb 14 04:36:16.115: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
Feb 14 04:36:17.115: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to down
Feb 14 04:36:19.792: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
Feb 14 04:36:20.795: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up

cisco Switch port configuration

interface GigabitEthernet1/0/1
description ***Primary SDWAN***
switchport access vlan 10
switchport mode access
!
interface GigabitEthernet1/0/2
description ***Secondary SDWAN***
switchport access vlan 10
switchport mode access

[Brash]

It looks like HA is constantly kicking in and  failing over the MX VRRP IP.

Are you seeing this on the dashboard?

What mode are the MX's in and what does the topology look like?

Are you seeing any issues with the uplinks?

[Ramakrishnan]

YES, i can see below error in dashboard after 10 seconds back to normal

Route connection change

peer_type: l3_vpn, connection_status: disconnected

Both MX's LAN ports are connected to one cisco L2 switch.

we didn't found any error in uplink ports

[MarcelTempelman]

Are these interfaces connected to the uplink ports or are these connected to the LAN-side of the MXs?

If you're seeing a lot of VRRP events keep an eye on the priority status. If you see prio 75 (primary) or prio 55 (spare) then you have issues with your uplinks.

https://documentation.meraki.com/MX/Networks_and_Routing/Routed_HA_Failover_Behavior

[Ramakrishnan]

these ports are connected to LAN side of the MXs.

no, i can see 255 on primary

[Bruce]

Not sure what your setup looks like, but make sure you only have one link from the switch to each of the MXs, and make sure there is no link directly between the MXs. Double check all your cables too, and make sure they are good.

[Ramakrishnan]

Both MX's LAN ports are connected to one cisco L2 switch. there is no direct cable connected between two MX's.

[rhbirkelund]

Are the MXs in VLAN mode, or are the using single LAN?

Consider perhaps configuring Spanning-Tree on the switch ports connecting to the MXs.

[Paul_L]

We have a similar problem on our MX250 HA pair - had to roll back from firmware version 16.16 to 15.55 due to VRRP failures - devices would switch over for no apparent reason with increasing regularity over some days. Ticket still ongoing.

[MayerS]

Hi,

same here since end of march (Case open 28.03.2022). Update from 17.3 to 17.5 and more. If the hot spare is available the traffic is poor and ping getting lost. shutting down on of the MX network ports all run fine. We have two redundant datacenters connected as metro datacenter in the city. Meraki support and we have done a lot of scans and traffic analyze. Meraki say it´s a problem of the Juniper switches. So after a lot of work we have moved the mx to one location and connected to the same Cisco switch. Result is the same if both MX are up and running the traffic is bad and pings get lost. The cisco switch that hold both MXs you can see the flapping of the both MXs NICs every 10-20 sec.

002207: Jul  5 07:41:20.351 MEST: %SW_MATM-4-MACFLAP_NOTIF: Host cc03.d907.a4d5 in vlan XXX is flapping between port Gi0/36 and port Gi0/41

Now is no Juniper between the devices all MXs are connected to the same switch and the problem still exist.

We are thinking again that the problem is on the MX Firmware ...

Any (other) ideas?

regards Stefan

[Paul_L]

Just got a reply to my ticket - hope it helps you

"Greetings,

The issue has been fixed by our Engineering team.
They informed that these fixes are present in MX 16.16.3 and MX 17.8

Could you double-check it, please?

Apologies for any inconvenience it may have caused."

[MayerS]

Hi,

thanks you for your quick responds Paul_L but we are running since some weeks on both MXs FW 17.8.

We have upgraded since march some firmware in the hope we can fix the problem but the trouble still exist.

regards Stefan

[Paul_L]

Fills me with confidence that my problem will be sorted then. Good luck with yours