MX250 -> MS425 Default Routes

[D52-MJ]

Hello-

I inherited two sites that were bound together over a vpn connection. Over the summer, we installed dark fiber to connect the two sites together. These sites were utilizing fairly old Cisco switching, but fairly new datacenter infrastructure and Meraki AP's. Eh, 2 out of 3. Not bad. 

Site A was 192.168.1.X

Site B was 192.168.2.x

Datacenter was located at Site A but the datacenter vendor started throwing in 10.x.x.x for the servers, iscsi, etc. in an effort to move to that particular structure. 

All the ap's and their respective clients are all in the 192's. There is a ton of legacy fire, phone, building automation, access control, etc. that is beyond replacing so the 192's have to stay for budget's sake.  I replaced the entire 100 meg and mix of gig switches with Meraki to go end to end including the MX over the summer with little to no problems. 

The MX is setup with 2 static routes from 192.168.x.x/16 and 10.x.x.x/8 to the next hop dist switch. The dist switch has the default route 0.0.0.0 back to the MX IP. All good. Fast forward. New ISP.    Change the IP on the MX interface and only the 192's will work, nothing on the 10.x network. I've rolled the entire network to the release candidates, I've 1:1 nat'd, etc. to no avail. I can setup 1:1 nat, shaping, flow preference, and literally anything I want on the 192 subnet in either building (except if it's wireless) and it works like a champ. Nothing changed except the ISP.   

I can direct connect to the MX (which is in the 10.x range) and can resolve some sites, but not every site. It's the strangest thing. Anyone ever seen this before? I'm a CLI by trade, I've run some packet captures, the equipment is less than 60 days old, and up until a few hours ago, not a problem at all. It's hard to blame the ISP when you can run a speed test on a workstation and it maxes out at 990 meg...  

Dist Switch has Layer 3 enabled, maybe 15 or so vlans. All seem to be functioning properly. As soon as I activate the old ISP, everything works as it should.  I've restarted DNS services, flushed cache, no content filtering enabled, no ACL, no NAT, No firewall rules. This is very vanilla. 

I've used Google DNS, my old ISP DNS, new ISP DNS, and everything else.  It just stalls loading "some" sites. Others load fine. But you can nslookup and resolve them via command line.  

[cmr]

What does the new ISO provide, just a tail or an active device?

If you plug a workstation into the new ISP and trace route to the internet what do you see, any 10.?

If you add the new ISP to the MX does it affect internal traffic or just internet access for the 10. devices?

Also why are you talking about replacing devices to change from 192. to 10.?  We were taken over about 13 years ago and had to renumber a few hundred devices, it wasn't that hard...

[D52-MJ]

It's a part ISP equation. ! part, the circuit provider installed a device, which terminates fiber, handed off single mode LR SFP to a state run entity's piece of equipment, which then hands us a multi mode SFP to our equipment.  

Unfortunately, you can not plug into the new ISP, it's a 10 gig SFP+ hand off. The closest you can get is the copper ports on the MX. 

No internal traffic issues. Just internet access. I think It's the states device not having a rule to send 10.x traffic, but I havent been able to get someone to get eyes on yet. The circuit provider is in a different subnet, and I dont have any other 10 gig equipment available to pull and test (I dont even know if it would work).  I'm just trying to save myself a headache of keep trying all these config changes when it feels like if the only change was the ISP, it has to be on that end. 

I'm aware of the difficulty level, I've done things like this before. But a lot of these systems are hard coded circuit boards that would require service calls to A/C, Door Company, and Fire Alarm vendors. They would have to track a lot of this stuff down that has been put together over the years and would be quite costly, especially with a new campus on the horizon. Is it possible? yes. At this juncture, things appeared to be operating just fine. So unless this appears to be the culprit, then there's no need to do this right now. 

I work in an area where budgets are stretched very thin. 

[DarrenOC]

Looks like you’ve answered your own question there with bringing the state run device into play.

If you run a trace route from either subnet does it hit an ip on the state run device? Therefore does it stop dead on the MX from the 10.x subnet?

[D52-MJ]

Unfortunately, a tracert from both the "working" and "non-working" site produces the exact same route success route hops.  The only difference is, the page doesnt display.  Quite baffling. I've updated the cases with Meraki and the ISP.