I'm having an issue with static routes on an MX showing down in the dashboard. We have an MX at each branch in NAT mode with multiple vlans. (It's acting as the branch router). Each branch also has a Cisco 3560x switch with a link to the MX that I use for routing to a remote datacenter. I've configured an uplink network between the MX & the 3560x, 10.0.0.0/30. The MX IP address is 10.0.0.1, the 3560x has 10.0.0.2. I've configured static routes on the MX for the remote datacenter subnets, 10.20.0.0/22 with a next hop of 10.0.0.2. I've also set the route to be active as long as host 10.20.0.1 responds to ping.
I have static routes configured on the 3560x for the branch subnets handled by the MX, and I re-distribute those static routes to the remote datacenter via OSPF. (So the datacenter has a route to the branch subnets).
This entire setup is working fine with the 3560x in place. We have the same step at each location (with different IP addresses) and I've had no issues with it for 18 months.
We're replacing the 3560x switch at each branch with a Meraki MS250. I ran into an issue with the first location where we replaced the 3560x. The MS250 has the exact same configuration as the 3560x, and the MX configuration hasn't changed. Now my MX routing table is showing it's static route to 10.20.0.0/22 (the datacenter) as down. From a computer on one of the MX vlans I can ping the MX interface 10.0.0.1, but I can't ping the MS250 virtual interface 10.0.0.2. Those 2 interfaces are directly connected (with the same patch cable the 3560x was using).
To troubleshoot the issue, I removed the condition on the static route configured on the MX "as long as host 10.20.0.1 responds to ping." I set the route to "always". The static route still shows down on the MX. But I can ping a host in my remote datacenter, (10.20.0.1), from a host on a MX vlan, and I can ping from a datacenter host to a host on the MX. I’ve also confirmed this in packet captures run on the MX port and the switch port connecting to the MX. Traffic is passing through the routing uplink. However, I still can’t ping the virtual interface 10.0.0.2 on the MS250.
If I turn off OSPF on the MS250 virtual interface for the routing uplink I’m able to ping that 10.0.0.2 interface, and the static route on the MX shows as up. But when I do that, the MS250 routes to the MX vlans aren’t re-distributed to the remote datacenter.
I’ve had a case open with support for 2 weeks now and I’m wondering if anyone else has run into this issue?
There is a chance the MX is caching the old ARP entry for the old switch. Can you give the MX a power cycle to make sure the only entry in the ARP cache is the new switch?
Can the MX ping the switch IP on the uplink? Is the uplink a VLAN interface or a WAN interface on the MX?
I've re-booted the MX, and even did a factory reset, and let it re-download it's config.
The MX can't ping the switch IP on the uplink either. The uplink is a vlan interface on the MX. And I have that vlan assigned to an access port on the MX.
The MS250 is running 11.30 firmware. The MX is running 14.40
I've also been able to re-create this issue in a lab environment.