Second attempt failed to upgrade MR52's, packet loss and slowness in switch after upgrade MAC Flap

[Thagaman]

Second attempt failed to upgrade MR52's, packet loss in AP's and slowness in switch after upgrade with MAC Flapping.  Everything works great on 25.13, but when I upgrade to 28.5, AP's show 100% loss when I ping them, and my Cisco 3850 switch runs super sluggish.  Tech Support helped me rollback to 25.13, but how to I get to a stable release?  The moment the AP's upgraded, and rebooted.  I see this in my Cisco 3850's, these mac flapping logs, and the MAC address is my MR52's.

Jan 10 18:26:08.134 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.2cb0 in vlan 72 is flapping between port Po24 and port Po1
Jan 10 18:26:08.571 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.30e2 in vlan 72 is flapping between port Po1 and port Po24
Jan 10 18:26:09.061 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.357a in vlan 72 is flapping between port Po1 and port Po24
Jan 10 18:26:11.991 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.22a0 in vlan 72 is flapping between port Po1 and port Po24
Jan 10 18:26:14.350 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.2cb0 in vlan 72 is flapping between port Po1 and port Po24
Jan 10 18:26:14.373 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.357a in vlan 72 is flapping between port Po1 and port Po22
Jan 10 18:26:14.375 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.30e2 in vlan 72 is flapping between port Po1 and port Po22
Jan 10 18:26:15.415 MST: %PLATFORM_MATM-4-QUEUE_OVERLIMIT: MATM dropped learn cache batch messages as queue limit has reached.
Jan 10 18:26:16.851 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.34f6 in vlan 72 is flapping between port Po1 and port Po22
Jan 10 18:26:22.065 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.357c in vlan 72 is flapping between port Po22 and port Po24
Jan 10 18:26:29.389 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.357a in vlan 72 is flapping between port Po22 and port Po24
Jan 10 18:26:29.443 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.2cb0 in vlan 72 is flapping between port Po1 and port Po24
Jan 10 18:26:29.622 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.34f6 in vlan 72 is flapping between port Po22 and port Po1
Jan 10 18:26:41.045 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.357a in vlan 72 is flapping between port Po24 and port Po22
Jan 10 18:26:42.033 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.357a in vlan 72 is flapping between port Po22 and port Po24
Jan 10 18:26:42.842 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.22a0 in vlan 72 is flapping between port Po1 and port Po24
Jan 10 18:26:44.041 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.32de in vlan 72 is flapping between port Po1 and port Po24
Jan 10 18:26:44.421 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.357a in vlan 72 is flapping between port Po22 and port Po24
Jan 10 18:26:44.579 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.2cb0 in vlan 72 is flapping between port Po24 and port Po22
Jan 10 18:26:44.615 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.34f6 in vlan 72 is flapping between port Po22 and port Po24
Jan 10 18:26:45.296 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.357c in vlan 72 is flapping between port Po24 and port Po1
Jan 10 18:26:47.057 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.32de in vlan 72 is flapping between port Po22 and port Po1
Jan 10 18:26:49.256 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.22a0 in vlan 72 is flapping between port Po24 and port Po1
Jan 10 18:26:53.641 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.3506 in vlan 72 is flapping between port Po22 and port Po1
Jan 10 18:26:59.503 MST: %SW_MATM-4-MACFLAP_NOTIF: Host e055.3dc0.357c in vlan 72 is flapping between port Po1 and port Po24

[Thagaman]

I have read other posts about MAC flapping, but there was nobody in the building, and no mac where roaming between AP's, and the MAC address is my AP.

[Brash]

Given what you've described, it's quite possibly a firmware.
But under the assumption it's not, if you follow the MAC flaps down through the port-channels and switches, which devices and ports to they isolate to?

[Thagaman]

The AP go from my Cisco 3850 to my Nexus 9k, and the MAC address I see in the logs is the MAC address of the MR52 AP.

[ww]

Are you using 1 or 2 lan ports of the ap?

[Thagaman]

2 Lan ports

[ww]

Maybe something is wrong with the bonding? I would move 1 or 2 ap's to a new network and try to get that working with meraki support.  try  fw upgrade in that new network and maybe disable 1 switchport to the ap

[Thagaman]

Thanks for the suggestion, I am  planning on removing the port aggregation from the MR52, and remove LACP from the switch.

[Thagaman]

I see flapping in 3850 and Nexus core switch.

2022 Jan 10 18:33:04 HQDC-N5672-A %STP-2-DISPUTE_CLEARED: Dispute resolved for port port-channel120 on VLAN0126.
2022 Jan 10 18:33:04 HQDC-N5672-A %STP-2-DISPUTE_CLEARED: Dispute resolved for port port-channel130 on VLAN0254.
2022 Jan 10 18:33:05 HQDC-N5672-A %STP-2-DISPUTE_DETECTED: Dispute detected on port port-channel110 on VLAN0112.
2022 Jan 10 18:33:05 HQDC-N5672-A %STP-2-DISPUTE_DETECTED: Dispute detected on port port-channel110 on VLAN0116.
2022 Jan 10 18:33:05 HQDC-N5672-A %STP-2-DISPUTE_DETECTED: Dispute detected on port port-channel110 on VLAN0071.
2022 Jan 10 18:33:05 HQDC-N5672-A %STP-2-DISPUTE_DETECTED: Dispute detected on port port-channel110 on VLAN0072.
2022 Jan 10 18:33:05 HQDC-N5672-A %STP-2-DISPUTE_DETECTED: Dispute detected on port port-channel110 on VLAN0076.
2022 Jan 10 18:33:06 HQDC-N5672-A %STP-2-DISPUTE_DETECTED: Dispute detected on port port-channel110 on VLAN0091.
2022 Jan 10 18:33:06 HQDC-N5672-A %STP-2-DISPUTE_DETECTED: Dispute detected on port port-channel110 on VLAN0081.
2022 Jan 10 18:33:06 HQDC-N5672-A %STP-2-DISPUTE_CLEARED: Dispute resolved for port port-channel110 on VLAN0071.
2022 Jan 10 18:33:06 HQDC-N5672-A %STP-2-DISPUTE_CLEARED: Dispute resolved for port port-channel110 on VLAN0072.
2022 Jan 10 18:33:06 HQDC-N5672-A %STP-2-DISPUTE_CLEARED: Dispute resolved for port port-channel110 on VLAN0076.
2022 Jan 10 18:33:06 HQDC-N5672-A %STP-2-DISPUTE_CLEARED: Dispute resolved for port port-channel110 on VLAN0081.
2022 Jan 10 18:33:06 HQDC-N5672-A %STP-2-DISPUTE_CLEARED: Dispute resolved for port port-channel110 on VLAN0091.
2022 Jan 10 18:33:06 HQDC-N5672-A %STP-2-DISPUTE_CLEARED: Dispute resolved for port port-channel110 on VLAN0112.
2022 Jan 10 18:33:06 HQDC-N5672-A %STP-2-DISPUTE_CLEARED: Dispute resolved for port port-channel110 on VLAN0116.

[PhilipDAth]

That sounds like an LACP issue, a failure to negotiate.

I don't know anyone who uses LACP with the APs.  It's pretty hard to get an AP to deliver even 1Gb/s of wired throughput.

I bet if you disconnected those second cables your life would become easier with no performance difference ...

[Thagaman]

thanks for the input, appreciate the help.  Meraki's website suggested configuration with a 3850 is LACP.  Do AP's do PaGp?