I accidentally found some undocumented MX behavior while troubleshooting a customer by using iPerf.
Situation:
You have SD-WAN tunnels (hub to hub in this case but it would work the same in hub to spoke) between 2 MX peers both using their 2 WAN ports. So in this case you have 4 actual tunnels. WAN-1 to WAN-1, WAN-2 to WAN-2, WAN-1 to WAN-2 and WAN-2 to WAN-1.
The idea is that you can only influence outbound traffic in your SD-WAN rules where you can set WAN-1 or WAN-2 as exit interface for a certain application. But you don't really have control if your traffic leaves WAN-1 on 1 appliance it would be received on WAN-1 on the other appliance. In case you are using private links or a private ISP WAN you would want this to be the behavior. The problem is that the documentation does not really reveal how in some cases traffic could be crossed between WANs.
According to my troubleshooting test it seems that the SD-WAN rules are not just outbound but bi-directional. So you can actually influence on what WAN your MX receives certain traffic.
The test:
I had an iperf server behind the remote MX and an iperf client behind this one.
The networks on one side are all subnets of 10.105.0.0/16 and on the other side are 10.106.0.0/16.
There are rules on each MX that force all private network traffic over WAN2 (by using custom policy network:10.105.0.0/16 and 10.106.0.0/16). So as expected when monitoring the uplink page on both MX'es you could see both WAN-2 traffic going way up during the test.
Then I added another SD-WAN rule on one side where I matched all traffic going to or coming from: TCP/5201 (default iperf port) to use WAN-1 instead. When I did the outbound test as expected both WAN-1 links were being used for that transfer. However when I did the inbound test WAN-2 was going up while WAN-1!! was going up on this side. So the remote MX peer was only matching the general WAN2 network rule while this side was matching the TCP/5201 rule and traffic was actually crossing.
Background:
I usually don't have customers that use datacenters to link up to Azure but rather direct internet connections. So I was not used to have many SD-WAN rules to be able to test this and the only real rules I can use is actual server usage between SD-WAN peers. The iperf test was a nice eye opener.
Takeaways:
So in complex SD-WAN environments where applications are used over the SD-WAN tunnels, be careful with your policies. If you don't want traffic crossing between your upstream providers which can cause extra delay or bandwidth issues be sure to double check both directions of your SD-WAN policies.
