cancel
Showing results for 
Search instead for 
Did you mean: 

Disabled switch (bad DNS) - no internet connectivity for all devices

Comes here often

Disabled switch (bad DNS) - no internet connectivity for all devices

Hey all,

 

so we have this problem in a network where we have Meraki MX and behind MX Meraki switches and behind those HP / cisco etc. switches and many Meraki and Ruckus APs in a hotel

 

This is the 2nd time this has happened, when a group of usually Americans come and start hosting meetings all the Meraki switches start going into Disabled switch (bad DNS) mode and what happens is that anyone connected to any switches or APs behind them has working DNS, so basically "no internet access", even though the actual VLANs are using different DNS addresses on different VLANs than the switches management etc.

 

Disabled switch (bad DNS) occurs every 10-15 minutes and stays for 2-5 in Disabled switch (bad DNS) mode and while in Disabled switch (bad DNS) mode, no DNS queries work so the customer says "no internet access"

 

Disabled switch (Bad Dns).JPG

Close up of the pumping DNS:

Disabled switch (Bad Dns)2.JPG

 

 

- I found nothing unusual from clients or traffic, except maybe 20-40 clients connecting to their VPN

- I have swapped the switched management DNS settings from ISP -> Google -> Internal but the problem persists

- I have switches management static and DHCP but the problem persists

- I have changed switch MTU from default to 1500 but the problem persists

- I found no other DHCP servers within the network

- I found no new devices connected to LAN via ethernet, so it must be via WLAN (Ruckus and/or Meraki MR)

- I disabled RSTP on Ruckus AP ports but the problem persists

- I have tried IGMP snooping and flood unkown multicast traffic enabled / disabled but the problem persists

- Firmwares are up to date and meraki support has gone through the settings and found nothing, only that it is "ISP problem", which it is not. Problem starts and ends as soon as the group starts working.

 

So what seems to happen is something is causing all of the DNS traffic to pump in 10-15 minute cycles and it seems to affect only up to switches (MS-220 series) (not MX) and everything behind switches. Management VLAN 2 (where the switches are) and also all traffic VLANs.

 

Has anyone come across this or anything like it?

10 REPLIES 10
Kind of a big deal ww
Kind of a big deal

Re: Disabled switch (bad DNS) - no internet connectivity for all devices

https://community.meraki.com/t5/Switching/Disabled-Switch-BAD-DNS/td-p/32382

 

so maybe check the logging of the other vendor switches for any clues. 

 

 

Comes here often

Re: Disabled switch (bad DNS) - no internet connectivity for all devices

All of the customers with problems we're directly connected to APs (Ruckus and Meraki MR) behind Meraki PoE switches, lets say that the HP switches etc. are behind a fiber link elsewhere and leave them for now.

We also checked that the IP settings were up to date locally on each switch without duplicates.
Kind of a big deal

Re: Disabled switch (bad DNS) - no internet connectivity for all devices

A long shot but perhaps this is a spanning tree issue and something is creasing the root.

 

Have you given whatever is the core switch in your network a low spanning tree priority, like 0?

https://documentation.meraki.com/MS/Other_Topics/Switch_Settings

Meraki Employee

Re: Disabled switch (bad DNS) - no internet connectivity for all devices

If the issue is reproducible in the environment,

The one of step to troubleshoot this issue is that taking packet capture to see where DNS query / answer is dropped between device reports the issue and the DNS server, because the warning message (Bad DNS) is shown up when the device is unable to receive answer from the configured DNS server.

 

As you may know, taking packet capture can be done on the Meraki dashboard from Network-wide > Packet capture > select target switch and the uplink port.

You can have a look if DNS query is sent from the device and answer for the query comes back properly to the device or not.

If DNS answer is not seen in the packet capture, moving this forward closer to DNS server end would be the idea.

 

Hope this process would help to find out the root cause of this issue.

 

Kind of a big deal

Re: Disabled switch (bad DNS) - no internet connectivity for all devices

@Marcelino although the DNS is failing - I bet it is not a direct DNS issue, but something else.

 

For example, DNS rate limiting some where, spanning tree issue causing packets not to forward, duplicate IP address knocking something out, etc.

 

What I'm saying is - don't focus too tightly on just the DNS.  Look wider for other issues.

Comes here often

Re: Disabled switch (bad DNS) - no internet connectivity for all devices

Yes, the main meraki MS220-48LP right after firewall is set up as bridge priority 0 - likely root. RSTP is enabled but while I was testing I disabled RSTP from Ruckus AP ports, problem still persisted.

Comes here often

Re: Disabled switch (bad DNS) - no internet connectivity for all devices

Problem start and ended with the group, I think they were using alot of VPN connections to their server.

We did a live packet capture with Meraki support, only pings were sometimes unable to reach DNS servers.

 

I dont think that DNS is the problem, its the cause. Also ISP checked their fiber connections and router and found no errors.

 

The thing is, what could cause it? Multible VPN connections from WLAN to the US causes Meraki switches to go bonkers?

Kind of a big deal

Re: Disabled switch (bad DNS) - no internet connectivity for all devices

Did you have spare Internet bandwdith at the time -  or did your Internet circuit get flat lined?

 

What model MX do you have, and what as the total number of clients you have?

 

If you go Organisation/Overview and select just the network for your appliance - what was the device utilisation like?

Comes here often

Re: Disabled switch (bad DNS) - no internet connectivity for all devices


@PhilipDAth wrote:

Did you have spare Internet bandwdith at the time -  or did your Internet circuit get flat lined?

 

What model MX do you have, and what as the total number of clients you have?

 

If you go Organisation/Overview and select just the network for your appliance - what was the device utilisation like?


We have a 500/500M fiber connection with abaut 297 client devices at the time using MX84 with balanced threat protection rule sets. Max peek being at 60Mb/s and usually below 20Mb/s.
Comes here often

Re: Disabled switch (bad DNS) - no internet connectivity for all devices


@PhilipDAth wrote:

Did you have spare Internet bandwdith at the time -  or did your Internet circuit get flat lined?

 

What model MX do you have, and what as the total number of clients you have?

 

If you go Organisation/Overview and select just the network for your appliance - what was the device utilisation like?


Also the utilization from MX peeked at 25%, mostly at 15% ish
Welcome to the Meraki Community!
To start contributing, simply sign in with your Cisco account. If you don't yet have a Cisco account, you can sign up.