Disabled switch (bad DNS) - only at offices with AT&T fiber

[Lonestarr]

We have 160+ offices around the country, all using the same basic setup - MX with MS switch(es) and MR AP(s).  No other hardware anywhere on our network, and every site is built using the same template so only the subnet info is different.

We are finding a persistent issue that only seems to be affecting sites using AT&T fiber as the ISP (we use several different ISP's based on whatever is available in the area).  Every once in a while, seemingly random, the switch will go into Disabled switch (bad DNS) state.  This tends to happen overnight when the office is closed, so it's not due to any kind of traffic or changes on the network.  It will stay like this until we reboot the MX - that's the only thing we've found that fixes it but it does fix it every time.  We've discussed this with Meraki support and their suggestion was to change our management vlan settings to use Google DNS (we currently use Umbrella).  The problem appears randomly so we can't reproduce it, but it did happen today and changing to Google DNS did not fix it.

The modem at these sites is the BGW320-505.  It does not allow a true bridge mode, but we have disabled packet filtering and enabled IP passthrough mode.  Even so, the DNS it gives the MX is 192.168.1.254 and apparently this cannot be changed on this modem.  Maybe this is causing an issue?

Has anyone seen this issue?  Is this a known problem with these AT&T modems, and if so is there a solution?  This tends to happen overnight so there's no users generating traffic that could be triggering the error state, and the same configuration at 100+ other offices never results in this problem, it is only the offices with AT&T fiber.

[Ryan_Miles]

Is it only the switches that do this? The MX and MRs never have the DNS issue? Do the MRs use the same DNS config and mgmt VLAN?

[Lonestarr]

The MRs show the same DNS issue at the same time as the switches, and yes they are on the same management vlan.  The MX remains up and green and never shows the DNS issue.

[Ryan_Miles]

Does the MX also use the same DNS servers or different than the switches and APs?

[Lonestarr]

I'm not exactly sure how to answer that.

The WAN link on the MX is set to dynamic, so that uses whatever DNS is assigned by the ISP - in the case of these AT&T sites that seems to be the provided modem/router, since the WAN IP address is public but the DNS address is 192.168.1.254.

The MX acts as a router and gateway and DHCP server for each vlan - data, voice, secure wifi, management - and each of these is configured to use Umbrella for DNS.

[Ryan_Miles]

Just trying to narrow down if it's just devices that point to Umbrella. Simplest test IMO would be set it to use google DNS and leave it that way for awhile. If the problem stops then Umbrella is the issue.

[Lonestarr]

I get that.  That's essentially what Meraki support told us before.  Problem is that we were having the issue at several sites every couple days for a couple weeks, then support suggested using Google DNS.  We set some of the sites to use google and left the rest on Umbrella, expecting it to happen again and then we could determine if that was the solution (ie, if the Umbrella sites went down but the Google sites did not).  But then it didn't happen again at all for about 2 months, so for the sake of maintaining consistency across our networks we changed them all back to Umbrella.  Today was the first time since then that it's happened, and it was only one site.

We have 160+ sites, and only about 10-15 or so use AT&T fiber.  All 160+ use Umbrella for DNS, but only the AT&T fiber sites have had this issue.  That's why I'm thinking it's more something to do with AT&T's modem/router than with the Umbrella DNS settings, but of coarse AT&T tech support is no help if you can even get ahold of someone who understands what you're trying to tell them.

[PhilipDAth]

I don't know anything about AT&T.

Are theree any firmware updates available for the AT&T modem?

[AY2022]

Hi All, 

Beginning to see this in our networks. 

Question, while solving the cause of the bad DNS, is there any way to prevent the switch to go into disable mode (assuming  that it did go into disable mode) and just continue running?