WAN 2 Failover on WAN 1 Quality degradation

[lpopejoy]

I have a situation where our Wan1 is having extended periods of very high latency (>500ms). The MX is not completely dropping offline, but the internet becomes unusable. The issue is happening due to a DoS attack (250 mb/s of UDP traffic coming in on port 80). I would like to hook up WAN2 to a 4g gateway that I have as a failover, however, I'm concerned that the MX won't fail over since it is still "connected".

Can I use performance classes to force a failover when latency exceeds 200ms? The wording on that page seems to imply that this only impacts VPN traffic, so I was unsure if this would help since we do not use VPN.

[PhilipDAth · Accepted solution]

You are correct.  Performance classes are only for failing over AutoVPN traffic.

I suspect the MX wont failover given the case you have listed.  During an attack you will need to manually tell it to prefer the 4G circuit.  You do this under Security & SS-WAN/SD-WAN and Traffic Shapping.

[lpopejoy]

@PhilipDAth I was afraid of that.  We could switch manually, but then thats taking the bet that the device will actually be able to download configs, which it may, but it will probably take forever.

[lpopejoy]

...and the attacks last beween 15-60 min at a time, so it isn't like we are talking about 8 hours of solid down time.  Which means by the time I access the dashboard, update, and the device downloads the config, the issue might have taken care of itself.... 🙂

[BlakeRichardson]

@lpopejoy  Is the circuit you are using have a dynamic or statically assigned WAN IP?  How much work would it be to request a new IP from your ISP and update your DNS?

[lpopejoy]

@BlakeRichardson 

...well... I've done that.  Had a static IP, dropped it.  However, the DHCP lease doesn't expire so it remains pretty "static".  Ive asked they do a force release to allow us to get a new one, but  they don't have a way to do that.  They suggest leaving the modem unplugged for 24 hours so someone else gets our lease.  🙂

...that aside, this is a targeted attack.  aka students goofing off and thinking this is the best use of their school's IT budget.  When I did switch from static to dhcp (and thus change the IP), it only "fixed" the issue for about a week.

[BlakeRichardson]

@lpopejoy  Being a school are you using cloud based email i.e. G-Suite or O365?

Can you get a secondary connection and route staff / server traffic through that and let students have the munted connection? 

[PhilipDAth]

Is this ddos happening from inside or outside your school? 

Are you hosting any public facing servers you could move to the cloud? 

[lpopejoy]

Outside the school.  A single external host is sending 250 mb/s of UDP traffic to our public IP.  The IP is owned by Digital Ocean, and I have contacted their abuse contact... so far nothing.  However, even after that is taken down, nothing keeps it from starting back up again from another source.

No public services hosted @ the school, so that isn't a concern.  We just need a reliable internet connection.

[lpopejoy]

@BlakeRichardson 

Yes, which is kind of my recommendation thus this post.

However, I wanted to keep more of a "primary w/ failover" model.  Otherwise, nothing will prevent a student from gaining access to a staff laptop at some point and doing a quick "whatsmyip" and starting the attack on the 2nd connection.  Won't take long for them to realize their DoS isn't working...

However if I leverage a failover connection, according to the comments on this post, it isn't going to fail over.  And that's a bit of a problem!

[jdsilva]

FWIW I have a feature request in to be able to set Internet failover between WAN interfaces based on % packet loss. In my testing the MX does not reliably fail over to the alternate WAN until >80% packet loss. 

I would encourage everyone to make wishes and also request this through their reps if this type of functionality is important to you. It might help my feature request become reality 🙂

[cmr]

If you have a spare port on the firewall, you could set it up with the IP of the DDOSer as that should stop your WAN interface responding to packets from that IP.  Just an idea and not sure how successful it might be as the route to you would probably still be saturated...

[lpopejoy]

@cmr  that's the issue, link is saturated.   We are already dropping the packets so no issues there.

[lpopejoy]

@jdsilva 

I agree....I made a wish this AM.  In my experience, those just go into a black hole.  

My 30000 requests for better real time visibility of client traffic has been ignored so far...