Seeing odd issues with SD-WAN for sites with LTE ISP

[DaBros]

So we have a customer in the retail space with 40-ish locations. All sites have MX-67, and we have a VMX-M in Azure and an MX-250 pair in their HQ. A handful of sites have LTE only as their ISP due to location / type of store setup. over the past several months, one at a time the LTE sites have started to have degradation on their communication to Azure, but not to the public internet nor to the HQ. 

We have packet loss in the 30 - 50 % range, to the point that their POS terminals at the sites are losing connection to Azure (where the control software lives) multiple times in the same transaction. However, there is no significant packet loss in the communication with the servers at the HQ, nor is there any issues with other cloud based software. Teams runs fine for Pete's sake.  Has anyone seen something like this before? 

thanks!

[alemabrahao]

Have there been any recent firmware updates?

[DaBros]

No. all of the MX's were on 18.107.2 well before the first site started to fail. we were on a VMX100 in Azure, and then swapped over to a VMX-M last week for licensing requirements, and the problem didn't change. (other sites started to get weird, but that's a different problem)

[cmr]

There was an issue with vMXs having significant packet loss in azure, @PhilipDAth had a way of fixing it I believe.  I remember it was something to do with how the networking was configured.

[DaBros]

Thanks!

@PhilipDAth I was searching your posts but didn't find anything related. do you have any pointers to check? 

thanks a lot.

[PhilipDAth]

Make sure the VMX is not deployed in the same Azure subnet as the servers it is trying to access.  This setup is the most common one to cause packet loss.

[DaBros]

They are on different subnets. the odd part is that its only the stores on LTE, but my assumption is that the LTE connection is a little less stable and so whatever is causing the packet loss is amplified. We are seeing minor intermittent loss on other sites, but nothing consistent.