Issue with MX250 connected to Internet through Layer 3 MS250

[KDL]

Hello all,

Hoping someone can provide some help on the following.

We have an MX250 with ONLY an Internet uplink connected to an access port on the MS250, which in turn is directly connected to the ISP via another access port - the MS250 is configured for Layer 3 and is managed from a connection to the internal Core. We are having issues with the MX250 not connecting to the Meraki cloud, with the dashboard indicating that it Has never connected to the Meraki cloud - the MX Local Status page shows that the MX has Internet access but cannot connect to the Cisco Meraki cloud - a packet capture on the switch port indicates that the MX can ping the DNS server, 8.8.8.8, successfully, and use it to resolve the domain, n1.meraki.com - the blanked out IP is the MX250 uplink IP

and the system LED goes through the rainbow colors. The packet capture also shows that a TLS session is started but the TCP session resets after an Unknown CA Alert - we did not see the MX attempt to use either of the UDP 7351 or TCP 7734 ports in the packet capture, and we think that these not be used to connect to the Meraki cloud anymore on later MX versions, but not sure on that - we are using MX16.16.

Just some further detail - The MX - MS link uses a transit subnet, which is an additional public IP block provided by the ISP. The MX uplink IP and the VLAN interface associated with the MS access port are both configured with an IP from this subnet, with the MS VLAN IP acting as a default gateway for the MX. The MS is directly connected to the ISP via another access port in a different VLAN, configured with the /30 subnet IP for the ISP and the default gateway set to use the ISP.

Any feedback appreciated if someone has had issues trying to use this topology or run into that Unknown CA Alert 

[jbright]

What happens if you connect a laptop to the same ethernet connection that you are trying to use for the MX WAN interface and assign the same IP address to it that you were trying to use for the MX? Can you reach the internet with it?

[KDL]

Hello jbright, thanks for your reply, yes, we tried that and we were able to browse the internet, etc. with the laptop configured and connected in place of the MX, as you mentioned 

[jbright]

Since you connected the ethernet connection to the laptop, I assume you are using a copper cable and not a fiber cable. There have been problems with some of the copper SFP modules working correctly in the MX250 and MX450 firewalls in the past. This was supposed to be resolved by at least the 16.16 MX firmware. Despite that, do you have another copper SFP module to test with? Another option would be two fiber SFP modules and a fiber jumper. Also, has this MX250 ever successfully connected to the Meraki cloud before?

[KDL]

Hello jbright, yes the ISP handoff is copper - we do have a second copper SFP but the link seems fine in that the MX can ping 8.8.8.8 across the link - we can change it though to be sure but it will be next week at this stage. The MX has successfully connected to the cloud when we connected the Ethernet uplink to the internal private data subnet of the Core switch, similar to how the MS250 switch is currently managed - this traffic gets to the Internet via a pair of MX100s to a different ISP.  

[KDL]

I should add that we have not connected the MX directly to the ISP yet to help eliminate any issues on the MS250 and confirm if still see the Unknown CA Alert

[jbright]

Connecting the MX directly to the ISP would be the logical next step in troubleshooting the problem.

If that works, you would need to look closer at the switch to see if something is being blocked, maybe with an access list . Beyond that, Meraki TAC would probably need to look at it too.

[KDL]

Thank you jbright, we will try connecting the MX directly when we are back on-site next week - the switch is just a couple of VLANs routing with no configured access-lists - I hope to open a ticket with Meraki tomorrow

[RomanMD]

Any static routes configured on the switch which might send the traffic to other destination instead to the ISP? Maybe the traffic is going via your internal network where you have an inline proxy/fw that decrypts the traffic?

On the other hand, I can confirm that there was a problem with MX250 which could not connect to the dashboard. This could be solved only after updating the MX to the latest version using copper SFP instead of 1GB FO SFP. 

[Stoerfaktor]

Why are you routing the MX traffic through the switch? Do you have any need to access the switch directly from the internet?

I would have tried to switch the traffic through your MS instead of routing it. That would also save you some public ip addresses, and you are managing the switch over a connection to the internal core anyway.

[KDL]

Hello RomanMD and Stoerfaktor, thank you for your replies.

RomanMD, just to confirm, we only have a static default route to the ISP on the switch - we can double-check next week that the traffic is going out through the ISP and not through the internal management network, by disconnecting the ISP. We do have copper SFPs at the moment and are running MX16.16 - jbright also mentioned these copper SFPs - do you recall if this was a link issue with the SFPs, or if it was passing traffic but just not traffic required to connect to the dashboard?

Stoerfaktor, as you mentioned, we have used the switch just as layer 2 in the past and that has worked fine - I neglected to mention that this is going to be a HA/Warm Spare setup, with 2 ISPs, so the /30 directly connected to each would not suffice - to get around that, as getting a larger subnet is seen as a a last resort, we did have spare IPs from another public block provided by each ISP and hence we are using those to route between the MX and MS, and having the MS route to the directly connected ISP subnet - at this stage we are only trying to prove that this topology works, MX250 to register through the Layer 3 MS250, with 1 ISP   

[KDL]

Just another update on this - we removed the MS250 switch and connected the MX250 directly to the ISP with the same result on the Local Status page - Internet access but cannot connect to the Meraki cloud - we left it for 15 mins and then decided to power cycle the MX250, and that seemed to kick start it to eventually registering on the dashboard.  We connected the Layer 3 MS250 switch in between the MX and ISP, same as the test setup that was failing last week, and this time the MX registered. We had another MX250 for a warm spare setup and it was exhibiting the same initial issue connected through the MS250 - this time we did not connect it directly to the ISP but just power cycled it and it registered OK - we did not power cycle the MX250's last week but we did factory reset the first one, which seems to have not provided the required kickstart. Speaking with Meraki support, they mentioned that the Unknown CA alert occurs sometimes if the ISP is doing deep inspection, which may cause it to use it's own certificate for the tunnel, and power cycling or factory resetting should resolve that - not expected to be a recurring problem once the MX250 eventually registers.