Final update and we have resolved the situation. This has been a right pain to fix but we got there in the end. Several things we went through: We tested that the traffic did indeed move between the site-to-site tunnel and WAN connections when adding exclusions. Packet capture verified this. We have a static IP provision in our Secure Connect environment but this only applies to traffic destined for services on ports 80/443 traffic. Outbound traffic to services running on any other ports will originate from Cisco's massive pool of dynamic IPs. This is a ridiculous oversight/limitation of the static IP provision cost. Meraki/Cisco should take note! The traffic for this troublesome app contacts an initial site for auth and license checking using HTTPS but then contacts the database instance on port 40,000-ish. This meant the traffic would originate from different source IPs, as per point 1. At this point, we considered that maybe the app's cloud server may be using session/IP-pinning of some kind and the disparity was causing the issue. The cloud service has some kind of load-balancers in front of the auth site, which changes the destination IP every few minutes. At this point we almost threw in the towel as the IP differences were at the third octet so excluding them would have been all but impossible without inadvertently excluding massive chunks of the internet. We couldn't perform DNS-based local internet breakout exclusions because we do not use the MX to proxy DNS requests. We use DNS This is how we fixed the issue on the client VPN but was impossible here. I managed to get the DNS-based exclusions working for a single laptop by altering a test VLAN to provide DHCP with 'upstream proxied DNS'. This worked immediately but is impractical due to the fact that the Meraki MX device uses the primary WAN port's DNS server to resolve, which in our case is Google's DNS. This means we lost local DNS resolution for the domain, which is a show-stopper. I don't know whether MX WAN interfaces would allow an internal DNS server (domain controllers) to be configured so didn't attempt. At this point we figured we'd give the Umbrella admin interface a deeper look in conjunction with the fantastic dbgview.exe from sysinternals. Once we established all the DNS names the app was contacting, we looked at the historic IPs associated with the DNS/URL entries, within Umbrella, and could see that although the IPs rotated quickly, there were only actually about 5 of them per service (about 17 total) and they'd been consistent for the last two years. We added all of the IPs to the tunnel exclusion and it finally worked. It's not ideal adding so many IPs to the list but given the massive change that would be involved with reconfiguring our internal DHCP scopes to use the MX for proxying resolution to our domain controllers, we'll be leaving it as-is. What a faff. Honestly, I'm not sure the Secure Connect solution is really worth it, given the limitation around static IP addressing and the extra complexity of internet breakout. Couple that to the complication of using the Umbrella agent on the local laptops, it makes DNS a much bigger pain to administer. Anyhow, all sorted.
... View more
