Replying to myself again. I've identified the trigger that causes the dataplane crash. I've passed the info on to support. ... View more

Unfortunately, it isn't going well. I spent a good deal of time analyzing the packet captures trying to identify a problematic flow that might be causing the problem but wasn't able to find one. Support is asking to troubleshoot live over the phone, which is difficult for me to support because each time the switch goes out it takes the rest of my network with it. I'm not sure what they'd see from the "backend" as they call it because the entire switch control plane dies, it won't be talking to the backend once the problem happens. ... View more

Replying to my own thread, again. I ran an additional test. I deleted and reconfigured the IGMP querier on another switch in the network. The same problem happened, 18 minutes after the IGMP querier was configured. During the test I ran a packet capture on a span port of the switch's uplink port. At the 18 minute mark five different control plane flows from the switch stopped responding:   1. A mtunnel link to the Dashboard went from a two-way traffic flow to one-way, with only traffic from the Dashboard being observed.   2. The second mtunnel link to the Dashboard went from a two-way traffic flow to one-way, with only traffic from the Dashboard being observed.   3. ICMP ping requests from my laptop to the switch started failing.   4. LLDP broadcasts from the switch stopped.   5. IGMP queries from the switch stopped.   All five of these flows stopped with 4 seconds of each other. The switch rebooted itself (presumably because the watchdog timer expired) around 7 minutes later. I've passed this all along to Support via the case I have open. ... View more

Replying to my own thread here. I've dug further into the surrounding evidence and it appears to enabling the IGMP querier for the first time on an MS120-8 switch is hosing up its control plane. Specifically:   1. The Dashboard reports "No Connectivity" for this switch when the problem occurs. This is an incorrect status as the MX67 which sits in front of the switch does not report the same. I believe "No Connectivity" is shown here because the control plane of the switch stops working. This makes it unable to properly send and/or respond to keepalives or heartbeats across its mtunnel to the Meraki Dashboard.   2. Once this problem occurs, DNS is no longer able to traverse the switch. I believe this is because the switch does some form of DNS snooping so that it can intercept queries to "switch.meraki.com" to be able to display the local status page. With the control plane jammed up/broken this DNS snooping fails. This is supported by the Dashboard status for all Meraki devices BEHIND this switch reporting "DNS Problem" during the same time period. I've also taken packet captures from the DNS server in the network at the same time and observed no incoming DNS requests.   3. There is a spanning tree topology change when this issue occurs. The other two switches in the network, which are only connected to this problematic switch, change from root ports to designated ports (as shown in the event log during the issue). This should not happen. I have specific spanning-tree priorities configured. This problematic switch should ALWAYS be the root switch. Spanning tree is a control plane process. When the control plane fails, spanning tree will also.   4. ICMP and SSH through traffic does traverse the switch during the issue. Presumably this type of traffic doesn't require any control plane processing so it is switched in hardware, as expected.   5. I suspect the unexpected reboot of the switch is because the platform watchdog timer embedded in the switch isn't being serviced because the control plane is hosed up. Once the control plane stops processing any traffic from the network I assume it also stops processing/servicing the watchdog as well. Based on the timing of events I suspect the watchdog timer is around 10 minutes.   6. I suspect other functions that require control plane processing are also failing though I haven't captured any data yet to be sure. DHCP (because of DHCP snooping), LLDP transmission, and the IGMP querier itself are probably also impacted by this when the control plane locks up.   For whatever reason the initialization of the IGMP Querier feature is locking/hosing up the switch's control plane. Unfortunately I don't have access to any platform logs so I can't tell if this is a CPUHOG issue, an input queue wedge, or something else bizarre. ... View more

How is this still not fixed?   Downstream VOIP RTP Packet Loss (MR42/MR42E/MR52/MR53/MR53E/MR84)   This has been broken for at least 18+ months. It makes Facetime calls absolutely unusable. The only workaround it to reset all DSCP values on the switch side to (presumably) prevent the traffic from landing in whatever queue on the AP is jacked up.  ... View more

@cuongct I'm not sure I follow your question. This particular bug was fixed in MR firmware 26.5. If you're running a newer version than that then you're probably encountering some other problem. ... View more

802.11r has security issues when using WPA2 PSK (pre-shared-keys). If you're using that mode, don't turn on 802.11r. Its primary benefit is speeding up the transition from access point to access point when using WPA2 Enterprise authentication. Proper network design and AP transmit power tuning is where you should focus your efforts if you're using WPA2 PSK. ... View more

@cmr wrote: The other thing to be aware of is radio power levels, smaller devices often cannot raise their power level above ~17dBm so to ensure that the AP can hear them when they can hear the AP you should keep maximum power restricted to this or below.  This. I've had really good luck with balancing the AP transmit level to the approximate level of my wireless devices. I run 15dbm on my MR42s and devices roam just fine from device to device. I'd work on surveying the AP power levels and coverage before bothering with 802.11r. ... View more

802.11r is unsafe when paired with WPA2 PSK (pre-shared-key). WPA2 Enterprise is safe to use with 802.11r, at least that's my understanding. ... View more

I have two Xboxes (Xboxen?) here. I did a port forward for each Xbox and the NAT test comes back as open. One is hard wired, an Xbox One X, and it' works fine. The other Xbox is an OG Xbox that uses wireless. Neither Xbox has any problem staying connected while playing games. The OG Xbox on wireless seems to have problems where it is no longer connected to Xbox Live when waking the system from sleep. I'm not sure what's up with that and I haven't spent any time debugging it. A reboot of the Xbox gets it back online.   I have a full stack here, 3x MR42, 3x MS120-8, and 1x MX67. ISP is Google Fiber.   What problems are you seeing? ... View more

It's explained here: https://documentation.meraki.com/MR/Group_Policies_and_Blacklisting/Applying_Policies_by_Device_Type#Client_Identification   ... View more

I'm sorry, but I don't understand this statement:   " Seeing it on my syslog server without any way to determine which rule it came from is not acceptable to us or our clients."   It says, in the syslog message, which pattern it came from. Sure, it doesn't say "Rule 15", but it does give you enough information to determine what rule it is from. It's not ideal, but the information is available to you. ... View more

Well, you could make that argument about anything it does. There's no way for you to get into the MX to see the actual ruleset that's configured. If that's a requirement for your environment you may want to consider a different product/solution that offers you that level of control / visibility.   Regarding your rules not generating hits. Are you saying in the Dashboard? If so, I've found those counters to not be real-time. The logging option on the rule is your best bet. However, I assume that is probably rate-limited to avoid causing a DoS on your own equipment by trying to log too much info too fast. Even regular big Cisco IOS/IOS-XE routers have a logging rate and queue limit. ... View more

Agreed, it doesn't tell you "rule 25 generated this log". You're going to have to determine that from the rest of the message contents. For example:   1589984219.325305969 mx67 flows src=192.168.1.82 dst=192.168.40.254 mac=AA:BB:CC:DD:EE:FF protocol=tcp sport=46928 dport=853 pattern: deny tcp && (dst port 53 || dst port 853)   Specifically, you need to look at the pattern. This will match what you've configured in your rules. The one from the above example is a Deny rule that blocks all TCP traffic to destination ports 53 and 853. Presumably you don't have several duplicate rules with the same matching criteria. ... View more

The only way to see what firewall rules are getting hit is to configure a syslog server (Network Wide ---> General) and turn on the "flows" option. After that check the syslog checkbox in the firewall rule. Sadly, there's no other way to do it. To figure out which rule is generating the syslog message you need to look at the pattern show in the syslog message itself, it should match whatever you configured on the rule. ... View more

This started working with both api keys late yesterday. I haven't changed anything on my end, it simply started working. I'm not sure what changed but it's good to go now. ... View more

I'd agree, if the server was sending a 302 back, but it isn't, it's sending a 404. ... View more

I believe this is an Apple-enforced restriction on the newer versions of iOS. See here: https://arstechnica.com/gadgets/2015/06/apple-to-require-6-digit-passcodes-on-newer-iphones-ipads-under-ios-9/ ... View more