Meraki

Crocker · ‎Oct 25 2023

Want to preface this by stating I am not terribly familiar with IPv6 in general... I have a handful of locations whose upstream ISP modem has IPv6/DHCPv6 enabled by default, with a 30-minute preferred lifetime and 60-minute valid lifetime. At these locations, when the modem hands a DHCP-NA or DHCP-PD renewal, there is a brief service disruption. I've seen two different behaviors: When the MX is processing the renewal, latency to the MX (and to anything behind the MX) will jump from 30-60ms up to hundreds or thousands of ms, then return to normal. When the MX is processing the renewal, latency to the MX (and to anything behind the MX) will jump from 30-60ms up to hundres or thousands of ms, one or two pings will drop, then everything returns to normal. There is no way to disable IPv6 functionality on the WAN interfaces of the MX. If I set static IPv6 addresses on the WAN interfaces, the MX still appears to process/handle the DHCPv6 renewals and experience the issues listed above. At some locations, I have either been able to login to the ISP modem myself and disable the IPv6 functionality, or have the ISP do it themselves; However, there are a couple locations where this cannot be done. All of the affected MX's are running MX 18.107.2; However, we first ran into this problem back when we upgraded from MX16.X to MX17.X - I don't remember the exact versions, but basically we upgraded from a version that had 0 IPv6 support to a version that at least had IPv6 WAN support. It was enough of a problem to roll back to the no-IPv6 software for several months. Am chasing up solutions with support, but curious if anyone else has run into this problem? For us, it manifested as VOIP call quality complaints from end-users.

Crocker · ‎Oct 16 2023

Done!

Crocker · ‎Aug 31 2023

We bumped into something like this at a subset of sites after upgrading from MX16.X to MX18.X earlier this year. At the most problematic site, it turned out that the upstream ISP modem had its DHCPv6 server enabled, with a 15 minute lifetime. Every 15 minutes, that lease would renew, and that would cause the MX to drop 1-3 packets. This was the same issue at the other sites that were affected, though the DHCPv6 server was handing out longer leases at those locations. Disabling that feature on the ISP modem was the resolution for this.

Crocker · ‎Aug 10 2023

Opened a case with Meraki support on Tuesday and didn't really get anywhere. Asked if it could be a shard issue, the rep was unable to check because an internal tool was broken...but was assured there were no large-scale issues. Reached back out today as this is ongoing, re-opened the ticket, and was informed that n426 (our shard) is experiencing degraded performance which is actively being worked on. This was after some testing via inPrivate, different browsers, off of the work VPN, and running a .HAR capture while attempting to claim yet another device for ~10 minutes.

Crocker · ‎Aug 8 2023

Something like that. We're swapping a mess of Cat2960's with MS120's at the branches. Basically got a room full of MS120's, our on-site technicians snag a few, head out to the branch to install the switch(es), send me the S/N's. One or two a day. When I claim those one or two S/N's, it's taking absolute ages for the claims to process. Historically, this has been instant/near-instant. Just curious if this is an us thing, a shard thing, or an everyone thing.

Crocker · ‎Aug 8 2023

Anyone else bumped into this? Taking upwards of 5-15 minutes to claim devices in the dashboard since early last week.

Crocker · ‎Jul 30 2023

Any Meraki folks haunting the forum who have any thoughts on this? I guess basically the question boils down to: How does a VPN concentrator determine which tunnel to use to send traffic to a destination at a spoke site with Active/Active AutoVPN enabled? Should I just expect to see occasional packets inbound over WAN2 for no apparent reason? The answer I got from support wasn't particularly clear, but I'd paraphrase it as "We don't know but we also don't think it's a problem"

Crocker · ‎Jul 27 2023

We're running 16.16.8 across the board, due to some unrelated issues regarding MX17/18's IPv6 support and some carrier modems with really low IPv6 DHCPv6. Unfortunately, that also means we don't have the option to select between graceful/immediate for WAN failover/fallback. I'm unsure what the 'default' is, in MX16. I did double and triple check our SD-WAN policy, but nothing in there looks out of place. Additionally, I'm using VOIP traffic as the example in my original post but I see other traffic (SNMP between our Solarwinds poller and the MX/MS's on-site) behaving the exact same way. That traffic absolutely is not affected by an SD-WAN policy, and should only ever traverse WAN 2 if WAN 1 goes down hard. When speaking with support yesterday, there was a little back-and-forth over whether this behavior was actually expected. I was told that the only real way to ensure traffic inbound to the spoke doesn't traverse WAN 2 would be to disable active/active AutoVPN. Wasn't particularly pleased with that answer, as it sort of dodges the crux of the issue.

Crocker · ‎Jul 26 2023

I wondered about this; However, I knocked WAN2 down for 10-15 minutes (assuming that was long enough to sever any flows). When I stood it back up, I immediately started seeing this behavior.

Crocker · ‎Jul 26 2023

Our topology is classic hub and spoke. We have 2 hubs (at separate datacenters), configured in one-armed concentrator mode. On the spoke side, we use full-tunnel active/active AutoVPN. Our spokes generally have a wired ISP connected to WAN 1, and an MG21 on WAN 2. Our spoke-side SD-WAN policy is simple. We instruct the local VOIP subnet to "use the uplink that's best for VOIP", which I understand is based on MOS. The rest of the traffic has no associated policy, thus should only traverse WAN 2 if WAN 1 goes down hard. What I'm seeing is that if I run a packet capture on the Site-To-Site VPN over WAN 2, there are a handful of packets sourced from our VOIP controllers traversing WAN 2. *Most* of a given flow is bidirectional across WAN1 (verified by running side-by-side packet captures), but occasionally a few packets come in across WAN2. I've double-checked the VPN statistics, and WAN1 is consistently at a MOS of 4.1, while WAN2 wavers a fair bit (due to bad cellular reception) from 2.5-3.8. As far as I can tell, I should never see any traffic over the Site-To-Site VPN cross WAN 2...However, I'm absolutely seeing this occur. What got me looking at this in the first place was a complaint about VOIP quality from a specific spoke. I found that disabling WAN 2 cleared up the issue (which again, is a bit cruddy due to bad cellular service). I know the 'fix' for the VOIP complaints is likely to either leave WAN 2 disabled (or adjust the VOIP policy to only shift traffic if WAN 1 is hard down); However...I'm stumped as to why *some* traffic from my hub to my spoke is crossing WAN 2. Any insights? I do have a ticket open with Support regarding this behavior, but thought I'd ask the community as well.

Crocker · ‎Jul 17 2023

Throwing my hat in for this one as well. An org-level event log would be excellent. Another that could benefit from org-level aggregation would be: https://api.meraki.com/api/v1/networks/$networkId/health/alerts

Crocker · ‎Jul 17 2023

This was Case #09382779

Crocker · ‎Jul 12 2023

Yup, all is good now. Was just a brief 5-10 minute disruption - thought it was a recurrence of the dashboard issues from late last week.

Crocker · ‎Jul 12 2023

Just started occurring within the past 5 minutes or so. EDIT: Apparently the fix was to post about it on the community forums. Am able to login and navigate now, but it's feeling sluggish.

Crocker · ‎Jul 7 2023

Throwing my hat in for the factory reset, that fixes all sorts of weird issues that I've bumped into. I've also seen a few rare situations where it takes way, way longer than usual for an MX to finally show as online/green in the Dashboard. Like hours.

Crocker · ‎Jul 7 2023

I had a support case open for this very issue. The last I heard was that the beta app had a resolution for the problem, but the rep wasn't able to forecast when this would be pushed to the live app and closed the ticket. I'd provide the ticket # but the dashboard's broken right now...

Crocker · ‎Jul 7 2023

This is a known issue that occurred at some point in MX17.X - not entirely sure which version, we jumped from 15.X to 17.X a few months ago and this started spamming our syslog. I opened a case with support, and to quote: "We do not yet have a full fix for this in versions of MX firmware that have support for IPv6 (namely 17 and above) but version 16 does not have this issue." The last update to that case was in February, haven't heard anything since.

Crocker · ‎Jun 7 2023

Got an update on my ticket yesterday that this issue has been resolved. After spot checking a few of my MS120's, it looks like this has been addressed.

Crocker · ‎Apr 21 2023

Had a little back and forth with support about rebooting switches and updating from MS14.X to MS15.X. Was able to show them a reboot has no effect, and that this issue is present on both flavors of MS software. The MS team is aware of the problem and are looking into it.

Crocker · ‎Apr 20 2023

Thanks for confirming it's not just something weird that only we are seeing. I've opened a case with support. If any Meraki folks are haunting the forums and want to take a look, this is case #09582507.

Crocker · ‎Apr 20 2023

14.33.1, no support case just yet.

Crocker · ‎Apr 20 2023

I've got an MS120 telling me it sees a MAC address on Port 57 (which is actually on port 9). Several other entries are supposedly on ports 52-54 as well. Thing is, this thing only has 48 ethernet ports + 4 SFP slots. None of the SFP slots are in use. Something buggy with the dashboard today? EDIT: This is happening on at least four different MS120's at separate locations. Neato.

Crocker · ‎Mar 27 2023

I think my favorite example of this so far is a "Dyson XBox 360" I found the other day.

Crocker · ‎Mar 15 2023

Agree. Have had to explain this to Security and Management teams several times over the years.

Crocker · ‎Mar 9 2023

Interesting. I *think* we tried those steps when I worked with support back in August, but it didn't work out. Don't quote me on that, though 🙂 I do see that the recent MS 15.20 RC release notes explicitly call out the exact issue we continue to experience with our MS120's: Ms120/125 known issues Ports with an odd-numbered MTU value fail to initialize (predates MS 11) Switches will never move a RADIUS server's connectivity status to available if it was ever lost resulting in all authentications being placed into the critical auth VLAN (present since MS 14.32)

About Crocker

Crocker

Community Record

Badges