DNS Latency/Loss

[Slobs2]

Hey all. I’ve been wrestling with something and I’d like to get some thoughts on it. I was getting reports from users of the network being slow. In my troubleshooting, I noticed that my MRs (MR56) were showing that clients were intermittently not getting DNS responses. Finding a dns perf tool, I verified very high DNS latency or just no responses to queries. I have two different ISPs connected to two separate MS125s acting as break out switches that go into a pair of MX450s in HA. When I switched to my other ISP, I again got a large number of no responses to DNS. I also did pings to various sites and servers but did not observe any loss or latency. After moving my perf tool from AP to then switch, to then MX and then to the break out switches, the DNS issue seemed consistent at each level. That is until I plugged the ISP directly into the MX, not going through the breakout switch. So it seems like removing the breakout switch may be the issue… The breakout MS is not showing low memory usage and does not seem to be under extreme stress. Anyone have any experience with this or have any additional thoughts?

[alemabrahao]

Some Meraki MS models have had firmware bugs that affect the forwarding of UDP traffic, especially at high rates or under specific conditions. I don't know your firmware version, but I would consider updating it.

I also suggest you open a support case.

[Slobs2]

They are on 17.1.3 but am going to be upgrading them to 17.2.1.1. I have a case too. 

[cqlm]

The MS125s should primarily be forwarding the DNS traffic to the Primary MX450 and then sending the return traffic to the client. To further investigate, I suggest taking packet captures on the MS switch to confirm they are properly receiving DNS traffic from both the clients and the DNS server.

Note: It may also be required to take captures on various stages of the network so the troubleshooting can be extensive and for faster resolutions, I suggest opening a support ticket.

[PhilipDAth]

There are a million different possibilities to this one.

Let's first start by only considering DNS.  DNS servers can return different answers depending on the provider you are using.  You have two different providers, complicating this further.

Are you using provider agnostic DNS like Google (8.8.8.8, 8.8.4.4) or provider DNS?  Provider DNS is more likely to get you into trouble when using dual providers.

If you deliberately take one provider offline at a time - does the issue still happen?  Does it by chance only happen when both providers are online, or only when one of the two providers is online?

[cmr]

I always recommend unmanaged switches as the breakout switches, if possible I'd suggest getting a couple of those (they are very low cost) and seeing if it makes a difference.  It would at least eliminate the MS125s as the possible cause.

The CBS110-5T-D is my go to model.

[Slobs2 · Accepted solution]

Just to post an update on this. After hours on with support, they determined it was caused by a "DNS snooping" feature on the switch. They disabled it on the breakout switches and almost immediately the DNS no responses stopped.

[cmr]

Interesting, and another reason to stay with unmanaged WAN switches!

[Tony-Sydney-AU]

Hi, @Slobs2 ! Thanks for sharing the solution.

Can you give us more details about the solution?

I'm curious about that DNS snooping feature.

[Slobs2]

No unfortunately. I guess all Meraki switches do a DNS snooping/analysis. When Meraki switches are being used as breakout switches, it’s too much for that service and causes DNS queries to go unanswered. Once they turned it off on those switches on their side, everything was totally normal. 

[Tony-Sydney-AU]

Thank you, @Slobs2 ! I was hoping to find some Public document about it based on your explanation.

That behaviour makes sense as you described. I was researching internally and found something.

Too bad I there is no Public document about it.

Thanks anyhow! It's good to know!