Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
About JHern
Community Record
12
Posts
16
Kudos
1
Solution
Badges
Sep 21 2022
1:39 PM
2 Kudos
Back to early in my career, I had a Mentor - Wally T. He taught me two VERY useful things in my career: "It is far easier to beg for forgiveness than it is to ask for permission." "It still sucks to beg for forgiveness. Make sure you have carefully thought through what you are about to do, so you can explain why you did what you did if/when needed." Thanks Wally!
... View more
One last update: We were also seeing the sporadic WAP disconnects from the Meraki Cloud issue on 28.6.1. Upgrading to 28.7 resolved the issue. No unexpected disconnects since the upgrade. I have confirmed with Support and my SE that the issue was specifically addressed in 28.7. Net: If you are on 28.X code, upgrade to 28.7.
... View more
I'd go ahead and open a case. There is a known issue of WAPs disconnecting from the cloud occasionally.
... View more
Just to make sure readers see this: Thanks to the efforts of our SE, fix for Layer 3 Roaming is confirmed in 28.7 and 29.X beta. Both should be available in the July timeframe.
... View more
Long time no see! Look me up - my cell number hasn't changed since we last talked. It was not Ziply. The ISP in question got the circuit in for us in a hurry, so I'm leaving them unnamed. That said, jumping from 27.7.1 to 28.6 on MR33s made a HUGE difference in network performance with a Layer 2 Bridging configuration. (From almost unusable to pretty good.)
... View more
I know it impacts MR56s, but I do not know if it impacts other MR WAPs. Speculation based on incomplete info: Probably platform-independant.
... View more
UPDATE: Fixed code levels confirmed. See below. There is a known issue with Meraki MR wireless code with the MTU changing when L3 Roaming. Using Ping on macOS, I am seeing an MTU of 1448 on WiFi. It should be 1500 to match the interface MTU on the client machines. I'm posting this to save others time and aggravation. Symptom Of The Issue Pinging on a SSID that uses Layer 3 Roaming. The WAP is actually telling us its MTU. Generate 1500 byte Ping packets - look at the 3rd line below % ping -c 2 -D -s 1472 www.boeing.com PING www.lb.boeing.com (130.76.184.142): 1472 data bytes 556 bytes from 10.34.130.34: frag needed and DF set (MTU 1448) Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 dc05 c58d 0 0000 40 01 9294 10.34.152.2 130.76.184.142 Request timeout for icmp_seq 0 556 bytes from 10.34.130.34: frag needed and DF set (MTU 1448) Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 dc05 a50c 0 0000 40 01 b315 10.34.152.2 130.76.184.142 --- www.lb.boeing.com ping statistics --- 2 packets transmitted, 0 packets received, 100.0% packet loss Lowering Ping packet size to 1448 - and it now works % ping -c 2 -D -s 1420 www.boeing.com PING www.lb.boeing.com (130.76.184.142): 1420 data bytes 1428 bytes from 130.76.184.142: icmp_seq=0 ttl=49 time=45.630 ms 1428 bytes from 130.76.184.142: icmp_seq=1 ttl=49 time=43.216 ms --- www.lb.boeing.com ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss Note: For the pings above, ICMP has a 28 byte overhead. (1472 + 28 = 1500 bytes.) Code levels impacted: Issue was found in MR 27 code and is still present in MR 28.6.1. Meraki Engineering is aware of this issue. Fix Incoming: The next 28.X firmware release is likely (but not yet confirmed) to have that issue resolved. No ETA for the next firmware release. Thanks to the efforts of our SE, fix is confirmed in 28.7 and 29.X beta. Both should be available in the July timeframe. Workaround from Meraki Support: Use Bridge Mode for now if it is not absolutely necessary to have L3 roaming. If you are having this issue, please open a case with Meraki Support so that they know you are impacted by the issue.
... View more
Labels:
- Labels:
-
Other
Meraki MR WAPs running 28.X code are supposed to have Path MTU Discovery (PMTUD) running on the management interfaces. However, it was not working for me. (There are multiple reasons why PMTUD may not work in all circumstances.) I'm putting this information, and the steps to troubleshoot, out in hopes of saving someone else the time and aggravation. This is what I was seeing for MR56 WAP connectivity to/from the Meraki cloud running 28.6.1 code. The 27.7.1 code did not show this behavior. The WAPs running 28.6.1 could not stay connected to the cloud for more than 5-10 minutes at a time. From the Dashboard, this is what we were seeing: So - What changed from 27.X to 28.X? This did: https://documentation.meraki.com/General_Administration/Cross-Platform_Content/Meraki_Device_to_Cloud_Connectivity_-_FIPS Relevant summary: Communication from the WAP to the Meraki cloud changed to SSL on TCP/443 in 28.X. Hints at the problem: Upstream firewall was showing no long-lived SSL sessions from the WAPs to the Meraki cloud. Sessions were lasting less than 90s. Also, the SSL sessions were being reset by the client, which while legal, is unusual. Packet capture in the upstream firewall showed a lot of TCP retransmissions of larger packets with the DF (don't fragment) flag set between the WAPs and the Meraki cloud. Finding and verifying the issue: Inserted a macOS laptop into the same VLAN/Subnet as the WAPs. Started doing Pings with a 1500 byte packet size to each L3 device upstream of the WAP, starting with the default gateway and ending at the Internet. ping -c 2 -D -s 1472 <insert destination here> Generates a 1500 byte packet - 28 bytes of ICMP overhead and 1472 bytes of payload. More help: https://documentation.meraki.com/General_Administration/Tools_and_Troubleshooting/Troubleshooting_MTU_Issues The Fix: It turns out that our new Internet link had a configuration problem. The MTU was misconfigured to 1496 bytes. Configuration was corrected to 1500 bytes. Once the MTU was corrected, WAP connectivity issues were resolved. In short, make sure you test your network's MTU size to make sure its 1500 bytes all the way from the WAPs to the Internet. Related threads: https://community.meraki.com/t5/Wireless-LAN/MR46-MR55-WIFI-6-AP-s-disconnect-from-dashboard/m-p/150403?utm_source=communitymembers&utm_medium=email&utm_campaign=immediate_general%27#M20883
... View more
Labels:
- Labels:
-
Other
Jun 1 2022
11:02 AM
There is a known issue with Meraki MR wireless code with the MTU changing when L3 Roaming. Using Ping on macOS, I am seeing an MTU of 1448 on WiFi. It should be 1500 to match the interface MTU on the client machines. Generate 1500 byte packets - look at the 3rd line below
% ping -c 2 -D -s 1472 www.boeing.com
PING www.lb.boeing.com (130.76.184.142): 1472 data bytes
556 bytes from 10.34.130.34: frag needed and DF set (MTU 1448)
Vr HL TOS Len ID Flg off TTL Pro cks Src Dst
4 5 00 dc05 c58d 0 0000 40 01 9294 10.34.152.2 130.76.184.142
Request timeout for icmp_seq 0
556 bytes from 10.34.130.34: frag needed and DF set (MTU 1448)
Vr HL TOS Len ID Flg off TTL Pro cks Src Dst
4 5 00 dc05 a50c 0 0000 40 01 b315 10.34.152.2 130.76.184.142
--- www.lb.boeing.com ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
Lowering packet size to 1448 - and it now works
% ping -c 2 -D -s 1420 www.boeing.com
PING www.lb.boeing.com (130.76.184.142): 1420 data bytes
1428 bytes from 130.76.184.142: icmp_seq=0 ttl=49 time=45.630 ms
1428 bytes from 130.76.184.142: icmp_seq=1 ttl=49 time=43.216 ms
--- www.lb.boeing.com ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss Note: For the pings above, ICMP has a 28 byte overhead. (1472 + 28 = 1500 bytes.) Issue was found in MR 27 code and is still present in MR 28.6.1. Meraki Engineering is aware of this issue. The next 28.X firmware release is likely (but not yet confirmed) to have that issue resolved. No ETA for the next firmware release. Workaround from Meraki Support: Use Bridge Mode for now if it is not absolutely necessary to have L3 roaming. As always, if you are having this issue, please open a case with Meraki Support so its known and tracked.
... View more
Update with SOLUTION to my issue. The issue I have been seeing is separate and distinct from the issue of WAPs disconnecting from the cloud occasionally. If you are seeing the connect and disconnect happen every 3-10 minutes over and over and over, read on. Actual problem found was that our new Internet link (about 3 weeks old) was misconfigured for an MTU of 1496. (Should have been 1500). As soon as the MTU was corrected, connectivity from the MR56s running 28.6.1 to the Meraki cloud became stable. Lessons learned: Meraki MR WAPs with 28.X code seem to REQUIRE an MTU of 1500 on the path between the WAP and the Meraki cloud. (I have asked if the WAPs are capable of doing Path MTU Discovery.) Check your MTU of your network, especially your Internet connection. The Ping command is handy for this since you can set the do not fragment (DF) flag to enabled and have a user-settable payload size. I hope this saves someone else a few hours of aggravation. One more thing: An Engineer from another firewall company took packet captures at the firewall upstream from the WAPs and before the Internet link where the issue was. He noticed that we had a lot of TCP packet retransmissions on larger packets with the DF (don't fragment) bit set. That was the hint that led to finding the MTU problem on the provider's link.
... View more
This is still an ongoing issue - See the thread on the subject at the URL below for all the details: https://community.meraki.com/t5/Wireless-LAN/MR46-MR55-WIFI-6-AP-s-disconnect-from-dashboard/m-p/149868#M20809. I'm running MR56 WAPs with 28.6.1 and they are unable to stay connected to the Meraki cloud for more than a few minutes a time. Going back to 27.7.1 is not a great option due to bugs fixed in 28 code.
... View more
Updates at bottom of posting. ------ There is definitely a known issue with some WAPs running 28.X code dropping / losing connection with the Meraki cloud dashboard. A change was made between MR 27.x and 28.x code on how the WAPs communicate to the Cisco Meraki Cloud to support FIPS compliance, and that appears to be the likely root cause. See FIPS links at bottom. If you are seeing this issue and have not opened a case, it's probably time to do so. My case number is 08050752 if you'd like to reference it. I have a pair of MR56 WAPs, one running 28.6.1 and the other running 27.7.1. The historical device data for connectivity to the cloud (below) speaks for itself. The Support Engineer I am working with says I am seeing it a lot worse than most folks. FIPS compliance Information: https://documentation.meraki.com/General_Administration/Cross-Platform_Content/Meraki_Device_to_Cloud_Connectivity_-_FIPS https://www.cisco.com/c/en/us/solutions/industries/government/global-government-certifications/fips-140.html https://www.cisco.com/c/dam/en_us/solutions/industries/government/security_certification/pdfs/mr-meraki-internal-compliance-letter.pdf https://www.cisco.com/c/dam/en_us/solutions/industries/government/security_certification/pdfs/ms-meraki-internal-compliance-letter-sw-version.pdf Lastly, a similar change also appears to have been made to the MS 15.1+ code and to the MX 16.4+ code as well. I have no idea if these new code levels will have similar problems, but a little extra caution is warranted for now. ------- UPDATE from Meraki Support: "I have narrowed down your issue to only the MR56s or Wifi 6 APs for now. The switches typically do not show this behavior and it is not expected you will experience this for the switches."
... View more
My Accepted Solutions
| Subject | Views | Posted |
|---|---|---|
| 2592 | Jun 4 2022 12:09 PM |
My Top Kudoed Posts
| Subject | Kudos | Views |
|---|---|---|
| 4 | 19838 | |
| 3 | 2685 | |
| 2 | 15861 | |
| 2 | 20309 | |
| 1 | 17879 |
© 2024 Cisco Systems, Inc.
