OSPF Routing to two default routes

[bsantomauro]

Hello all,

So I have an interesting situation I am working on. I work for a school district and we have 4 schools (a HS, MS, and two Elementaries). I am attaching a pdf network map with this post to try and assist with what I am about to describe. The two Elementary schools each have a 10g Fiber connection back to both the HS Core and MS Core. HS/MS cores are MS425-32's and the Elementary cores are MS425-16's. We have two internet connections in the district. One in the HS and one in the MS. These internet connections connect into a MX250 that is in each of these buildings. The MX250's connect back into the respective MS425-32 Core stacks. We have OSPF routing enabled which is how we are routing all of our VLANs between all the buildings.

What we are trying to accomplish is to have OSPF route our traffic out both Internet connections to sort of load balance our Internet traffic so we are utilizing both ISP connections at the same time. I know this isn't true load balancing. Now so far we have been able to get most of the way there. Both the HS and MS Cores have a static default route whose next hop ip points to the firewall in that building. Those static routes are advertised over OSPF and preferred over OSPF. Doing this caused the HS Internet traffic to go out the HS ISP and the MS Internet traffic to go out the MS ISP. The Elementaries seem to be evenly split using OSPF with one going out the HS connection and one going out the MS connection.

The issue we ran into is when we did some failover testing. We pulled the connection from the MS Core to the MS MX250. The HS continued out the HS connection and both Elementaries adjusted to route out the HS connection for Internet traffic. However, the MS just lost internet access and never routed out the HS connection which is what we expected to happen. What was odd is that the switches themselves seemed to adjust their routes because they retained their connections to the Meraki Cloud controller. The only way to fix this was to change the default route on the MS core to say "No" for preferring the default static route over OSPF routes.

I am trying to figure out a way to make this automated so that if one of firewalls goes down, connection to our cores goes out, etc. everything automatically fails over to the other functional default route. Has anyone here ever configured anything similar to this? Am I missing something here in my configuration that could be causing this? I am going to open a TAC case next week during Winter break to try and troubleshoot this but I figured I would ask here in case anyone else has dealt with anything like this.

I know this was a lot of text and I apologize. I appreciate any help or suggestions you might have. Thank you for your time.

[PhilipDAth]

>What we are trying to accomplish is to have OSPF route our traffic out both Internet connections to sort of load balance 

I would not do this.  You can use dual routes to use one or the other - but load balancing across two different sites using two different public IP address ranges is likely to break things.

Some things (for example Microsoft Teams) expect to see all of a user's traffic coming from one IP.  This kind of config could make one stream appear to come from one IP address and another stream come from a different public IP.

You'll get sick of people ringing you complaining about some app not working, or it working one day and not the next.

[bsantomauro]

Good morning Philip,

Thank you for the information. Unfortunately, my hands are tied here. We had someone come in and do a network upgrade over the past year and we ripped out our old Alcatel Switches for Meraki, Aruba AP's for Meraki, and our one Palo Alto for two MX250's. During this project it was mentioned that we were going to be running two internet connections across two of our buildings so the district is expecting us to be utilizing these day to day. I can't really go back to them and say this can't be done so I have to try and work with what I got to come up with the best solution possible. Thank you again for your suggestion.