Layer 3 Roaming Test - Unreachable AP's

[taugust04]

Hello everyone.  My apologies if this ends up being a duplicate post - my first post disappeared for some reason.

I'm having an interesting issue with a variable number of AP's failing the "Layer 3 Roaming Test" on our primary SSID.  We just deployed a full Meraki network (MX, MS425, MS250, MR30H, MR46) to our university's Residence Hall network.  We have about ~675 AP's deployed, and, the SSID in question is configured for Layer 3 Roaming.  We have a VLAN per room, setup with VLAN tagging, so students get a "home" VLAN when they connect their devices in their rooms.

In any event, when we run the Layer 3 Roaming test, a random number of AP's, between 12-34, are "unreachable".  If you run the test 5 times in a row, you can get five different numbers of AP's that are unreachable.  It's the same ones over and over, but sometimes they get removed from the list, and sometimes re-added.  If you check each individual AP in the dashboard that is "unreachable", they are online, fully functional, and have no issues.  Nothing of interest in the event log.  The APs are already on switches that have AP's connected that are reachable on the Layer 3 Roaming test at all times.  So far for troubleshooting, we've restarted AP's, re-tagged the AP's, re-configured switch ports they are connected to, and deleted and re-added the interfaces they are using in the dashboard.  So far none of those steps has resolved the issue.

We've opened a TAC case, but was hoping for some feedback on how to proceed on my own as well.  This one has me really stumped, and I've tried to do some research on "Layer 3 Roaming Test" failures and haven't found much.   I'm open to any suggestions - and let me know if I need to provide any other information that might help the community better understand this problem.

[PhilipDAth]

Are the switch port configurations all the same?  You haven't got any switch port MAC address limits?

The APs need to be able to talk to each other using their IP address.  Anything showing up on the L3 gateway being used to route all this traffic?  Anything showing up on the switch logs that the APs plug into?

[taugust04]

Hi Philip, thanks for your reply back.  To answer your questions:

- Switch port configurations are exactly the same for all the A/P's that pass and fail the test, in comparison to others on the same switch.

- No switch port MAC address limits.

- Switch logs are clear of anything that points to an issue when I run the Layer 3 Roaming Test on the SSID.

- Layer 3 Gateway is clear.

Thankfully one of the A/P's showing up on this list is one of our test A/P's located in my office.

I don't mind trying to do a packet capture to try and see where the problem might be, but I'm not sure where exactly I should be doing it on the network and what I should be looking for...

Thanks!