Internal DNS server for proxy to upstream DNS

[JamesHammy]

Hi folks,

Bit of an odd one. We use SecureConnect and Umbrella SWG in a multi-site MX environment. All local LAN/WAN traffic traverses the SecureConnect tunnel but for reasons I won't bore you with, we need to exclude certain domains from the tunnel and breakout locally. Because we use our local DNS servers (AD-integrated DNS) for internal resolution, we can't use DNS-based VPN exclusions on the MX.

Pre-req: We run a local Microsoft domain and I need to retain local DNS resolution to access domain resources.

I need to set some of our internal VLANs' DHCP config to have the MX105 perform the initial DNS resolution. The only way I can see this is possible is by configuring the DHCP scope's DNS servers as 'proxy to upstream DNS'. Having read a bunch of Meraki KB articles, it's clear that this 'proxy' mechanism works as follows:

1. Endpoint queries the MX 'DNS server'
2. MX forwards request to the local LAN interface (we have several LAN interfaces so not sure which it would even use?)
3. MX's local LAN forwards request to the primary WAN uplink's DNS servers
4. Endpoint gets its DNS query resolved

But here's the challenge. Our WAN interfaces use external DNS servers so that we can retain uplink health monitoring capabilities (we use both WAN interfaces on independent leased lines).

If we put aside losing WAN monitoring, is it actually supported to set our internal DNS servers on the WAN interfaces?

Any help would be great!

This is the article I've been referencing:

• https://documentation.meraki.com/SASE_and_SD-WAN/MX/Design_and_Configure/Configuration_Guides/DHCP/C....

[ww]

Maybe you can achieve/configure local dns resolve from the mx itself.

https://documentation.meraki.com/SASE_and_SD-WAN/MX/Operate_and_Maintain/How-Tos/Local_DNS_Service_o...

[JamesHammy]

Thanks for the link. I actually read through that document too but the problem is, I don't want to run a custom DNS service on the appliance itself. We'd have to manually manage the entries, which would be totally impractical in a domain environment, with hundreds of laptops that all use dynamic DNS updates. I just want the MX to forward the requests to our internal DNS servers.

[RWelch]

The WAN DNS settings are intended for public or ISP-provided DNS servers, which are required for proper MX operation, including Internet connectivity checks, Dashboard communication, and VPN registration.

Using internal DNS servers on the WAN interface can disrupt these functions.

[JamesHammy]

Thanks and that was exactly my concern. In my mind, especially given Meraki devices are 100% cloud-managed, WAN interfaces should only ever be configured with external DNS addresses.

The biggest challenge with proxying DNS requests from the MX, where we must retain local resolution for the domain, is the following line in the guide:

• Proxy to upstream DNS
  Clients will send DNS requests to the LAN interface of the MX, which will then proxy those requests to the DNS server(s) configured for its primary Internet uplink.

Doing the above completely sidesteps any local resolution so the machine will effectively fall off the domain. If we were pure cloud and had no domain, this whole problem would go away.

[RWelch]

To retain local DNS resolution for domain resources, configure your DHCP scopes to hand out your internal DNS servers directly to clients. This allows clients to resolve internal resources while still using public DNS for Internet-bound queries. 

[JamesHammy]

Yes, we've always had DHCP hand out our internal DNS servers to clients and this works perfectly for normal use.

Our problem is that I cannot do DNS-based VPN tunnel exclusions on the MX devices, unless the client initially uses the MX to query DNS. Obviously, once our client has queried our internal DNS server, the actual internet traffic sails through the MX as IP addresses and ports, so it can't see the domain names or URLs. This means unless we put IP-based tunnel exclusions on the MX, which would be impossible to manage for our use case, all internet traffic hits the tunnel.

I suspect the fact that the MX can only do layer-4 inspection is why we have such a problem. Layer-7 would obviously mean the traffic's domain name / URL woudl be sniffed out and rules applied?

I suspect 

[ShaunB93 · Accepted solution]

Assuming your MX's are the L3 gateway between the client devices and internal DNS server(s) - Wouldn't the DNS query and response to/from your Internal DNS server(s) still be observed by the local MX and therefore qualify for inclusion in DNS-based VPN exclusion rules?

As an example, if we wanted to exclude www.google.com from the VPN full-tunnel to Secure Connect:

1. Client PC makes a DNS query to for www.google.com to Internal DNS server
2. DNS query transits local MX where VPN tunnel exclusion is configured for FQDN www.google.com

3. Internal DNS server responds to query with IP for www.google.com (let's say IP 1.1.1.1 is returned in the query response)
4. DNS query response transits local MX where VPN tunnel exclusion is configured for www.google.com - local MX observes 1.1.1.1 provided in query response

5. Local MX adds 1.1.1.1 to VPN tunnel exclusions based on observing the DNS query and response

The DNS query and response must not be encrypted otherwise the local MX will not be able to see query and response details within the packets

A quick method I use to validate if DNS-based exclusion rules are working is to perform a traceroute to the FQDN after the first DNS query is made - you should see the trace following a path into the local WAN provider rather than into Secure Connect

I also believe that Meraki support are able to see/query the IP's that are added to VPN full-tunnel exclusion list by DNS-based rule configuration 

[ShaunB93]

Also worth mentioning in case you are in this scenario.

If your client machines are running Cisco Secure Client with the Umbrella module, it is possible that your internal domains are configured within Umbrella as per Cisco Security

In this case, DNS queries for configured Internal Domains are sent locally, all other DNS queries are sent directly to the Umbrella resolvers - these queries will be encrypted by default and thus the local MX is not able to see within the packets. The solution we use here is to use the L3 outbound firewall rules on the local MX to block encrypted DNS from the clients to the Umbrella resolvers whilst also excluding the Umbrella resolvers from the VPN full tunnel. 
The client will fallback to sending unencrypted DNS queries to the Umbrella resolvers and the local MX will be able to observe the query and response for consideration of DNS-based VPN full-tunnel exclusion rules. 

[JamesHammy]

@ShaunB93 This is an interesting setup too. We do indeed have internal domains excluded so while on our corporate network at all offices.

Sounds like because the Umbrella agent is enrypting any external queries, that's why they're not being captured and evaluated. And the only DNS queries hitting our internal DNS servers and purely for domain.local requests.

Next step is to try adding a DNS-based exclusion and then accessing the site from a device that has the Umbrella agent completely disabled. That way the DNS query will be unencrypted regardless of destination and my local resolver will be used for all queries. Which the MX should see.

[JamesHammy]

@ShaunB93 Thanks for this abd apologies, had a decent break from work over Christmas and now catching up.

We do exclude internal domains (all .local) from Umbrella client queries so anything domain-based always hits the local DNS servers for resolution.

Your note about DNS queries for external domains is interesting and yes, those queries are obviously encrypted so the MX couldn't sniff out the DNS-to-IP mapping for DNS-based traffic exclusion.

So, for any offices where unencrypted DNS queries actually traverse the MX (which will be the case for all our satellite offices), to our local DNS server, any DNS-based exclusions should be evaluated and work?

I've just tested adding in a DNS-based exclusion to one website, checked that our local DNS (at another office) is resolving the query and then accessed the site but it doesn't work. When visiting the site, the connecting IP address still shows as our Secure Connect (static) WAN IP and not the local ISP (static) WAN IP on the MX.

[JamesHammy]

@ShaunB93 You were right on the money with the encrypted DNS queries for external resources and the local resolver only handling resolution for internal requests.

I did the following:

1. Created a DNS-based VPN exclusion for www.whatsmyip.org
2. Accessed the site and verified that it showed Secure Connect IP (which it did)
3. Disabled the Cisco Umbrella Agent Windows service on laptop (which reverts all DNS queries to plain text on local resolver)
4. Accessed the site and verified it showed local WAN IP (which it did)

Result!

Rather than mess about with layer-3 MX firewall rules to revert all DNS queries to plain text, while in the offices, I figure a better way to solve these few exclusions is to instead add the domains to the 'Internal Domains' list within Umbrella Admin console, whch will send them (plain text hopefully) to our internal DNS server.

Thanks for your help!

[ShaunB93]

Nice one - sounds good!

Also worth adding that if your machines have SWG agent enabled whilst on-net(in office) you also need to add the bypass to External Domains list in Umbrella otherwise, whilst the DNS query will go locally based on the local domain entry, the HTTP/S connection still still establish via the SWG via the agent

[JamesHammy]

Good call. I'll add that as well. Funnily enough, we don't have the SWG agent enabled currently (future project when we start retiring client VPN) but it is enabled on mine as a test.

Thanks!

[JamesHammy]

Gah. Even with SWG agent disabled I'm now seeing strange behaviour.

If I browse straight to the excluded site, the traffic goes via SecureConnect. No matter how many times I refresh, or how long I leave it, it's always egresses via SecureConnect.

If I then ping the site (or tracert, which also pings) I can see the traffic egressing the WAN interface and the website then updates and shows the WAN IP address.

It's almost as if some traffic, other than HTTP/HTTPS, needs to pass through before the exclusion kicks in.

If I then flush my DNS cache, the traffic reverts back to egressing the SecureConnect IP, until I ping the site. Then it's back to the local WAN IP.