Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Register or Sign in
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
About Yellow_Tang
Yellow_Tang
Here to help
Member since May 30, 2023
Apr 4 2024
Community Record
6
Posts
0
Kudos
0
Solutions
Badges
Apr 1 2024
12:51 PM
I didn't catch that you were doing this ONLY during upgrades. I disabled one downstream port on every switch so we no longer have redundancy at all and it fixed the issue. I cannot reenable LACP though, or the problem will come back. I'm waiting for an upgrade past 15.21.1 to be stable in hopes that they address cross stack LACP. Of interesting note, I think if one side is Cross Stack, and the other goes to a single switch it's still okay. It's only if the redundant links are split across stack members on both sides. Thank you!
... View more
Jun 12 2023
8:44 AM
So,.. this got allot worse. I started noticing that some internal IP addresses would randomly not be reachable from certain machines on the network, but other machines connected to the same switch were just fine. I could even unplug a machine from a port, plug a different computer in, and it would start working. It was very intermittent, and the same machine that wasn't working yesterday would suddenly be working today, but another one wouldn't be. It's worth noting that the Cohesity VIP might not be pingable, but half the node addresses on the same subnet would respond, and half wouldn't. Which ones did and didn't was not consistent across different workstations either though. The only thing that was consistent was that all virtual computers always had a perfect connection to everything. I suspect that is because LACP is not setup between that switch and the core. As you can imagine this was causing some bizarre and difficult helpdesk tickets with glitchy behavior like this, and those poor guys have enough to do. I ended up nailing down a computer that couldn't ping our Cohesity. I got a persistent ping going then I disabled the cross/stack LACP by disabling one of the ports in the 2 port aggregate link. It immediately fixed that computer and a couple others that were having trouble in the same stack. Every one of our buildings has at least one LACP link to an IDF over SMF which spans 2 switches at the core, and 2 switches in most IDF's. Right now to keep the lights on I'm working on disabling LACP everywhere. I'd love to hear from people if LACP into a single switch is working? Or is LACP bad in this version? Thank you, Yellow_Tang
... View more
Jun 4 2023
3:31 PM
Thank you everyone for posting your experiences here! Unfortunately I will miss the 14 day rollback window due to travel, but if you go forward with that South Paw, I'd love to hear how it goes. Meraki support said there is absolutely no way to roll back after 14 days. I have not removed LACP, and I've noticed that I'm getting almost no UDLD errors now (none since the 24th actually), plus the switches have been fairly stable (Knock on Wood). I have my fingers crossed that this all works itself out. I've never heard of a firmware upgrade needing to break in,.. but maybe something still wasn't finished? I can only guess and hope lol. Thanks again! And if anyone else has experience with this please let me know, Yellow Tang
... View more
May 31 2023
10:31 AM
Thanks for the reply Brash! I had considered those known issues, but we were on 14 code before and didn't have this, I would have expected it there as well since one was always present and the other was from code 10. The other thing that is puzzling is that these errors aren't coming up when something is changed or rebooted. The switch is running just fine for a few days then I get a short burst of these errors from across campus, the errors come up and clear in a short time, but it seems to come in waves. Currently I haven't had a UDLD error since May 24th, but I had it on 6 completely different switch stacks, it occurred and cleared on all of them all within 10 minutes. Then on May 22nd I had another wave of these errors, all happening and clearing within about 25 minutes. and this was in 14 completely independent stacks and buildings. I'm not sure, I'm hoping it stops happening though. A big part of me was starting to wonder if one of my core switches didn't upgrade correctly. Yellow Tang
... View more
May 31 2023
10:24 AM
I had something similar happen with the same code upgrade. Half of our DHCP was handled on the core switch, and half our DHCP was handled by a DHCP relay to a server. Before the upgrade this worked great for years, after the upgrade this completely failed. I troubleshot with Meraki for some time, tried relaying to our MX instead and no dice. Eventually we followed the packets around and found that the request was coming in but the relay was not sending the request back out, it was just dying in the switch. In my case a reboot of the core stack after hours resolved the issue. I'm also seeing the UDLD errors on my fiber links that connect the building stacks back to the core. Not all of them, and not all the time, but every now and again I will get a burst of UDLD errors that present then clear in less than 2 minutes. Meraki wants logs of the event happening to proceed with troubleshooting but I'm unable to get that because of how quick the event is. Yellow Tang
... View more
May 30 2023
6:55 AM
I'm working on a challenging issue with support right now. We have a somewhat complex network, with about 65 switches, multiple stacks in 30+ buildings all interconnected by 10G single mode fiber. We have a pair of MS425-32's at the core, and every link out to the and MS350-48FP stacks in the buildings is LACP spanning the two core switches. All inter-vlan routing is done at the core. We were running 14.33.1 code and we went to 15.21.1 code because it attained the "Stable" rating. I upgraded remote sites first because they are simple with only one or two switches. Everything went smoothly so I let our main campus upgrade. That one went so bad. Half of our switches went down and never came back, stacks were half alive with some switches working and others not. The core looked green, but Core 1 turned out to not be working once the network was put under load. I tried to reboot it and it went down and didn't come back. Unplugging the stack completely and plugging it back in got it to boot again. That became the fix for every switch we have. Once force-rebooted I thought we were good, but now I have a couple switches that have dropped out of their stacks, then hours later they just pop back in. From the local status pages other switches in the stack show them as there, and the ports appear to be up, but the switch is gone and the light is red. In most cases the ports in the switch don't work. Also if I reboot the core some of my stacks will not reconnect to it until they are also rebooted. My other weird issue is that I'm now suddenly getting UDLD errors on my LACP links heading back specifically to Core 1, Core 2 never has an issue. This could be the stack acting up, or LACP acting up, I'm just not sure. I'm trying to revert to 14.33.1 code now, but I'm curious if anyone else has had issues anything like this on 15.21.1 or any 15 code? Thank you, James
... View more
© 2024 Cisco Systems, Inc.
