multiple stacks same network issue

[leadtheway]

Ok so heres a weird deal. First a little background.. Have two parallel networks with identical address spaces. One is a 3rd party vendor managed that is going away to be replaced with all meraki.  Theres also a 3rd party company that has equipment in between the two.  Several vlans currently using the MX100 for dhcp that only really need internet. The data vlan (which is 101) still resides on the old network. Can't use meraki to route it or dhcp/dns because the servers still reside on old equipment. As a workaround, we have been using the 3rd party by having them create a access port on that vlan, then we plug that into our switch stacks and then assign the vlan to the ports we need and all is well.  One site is proving to be difficult. Attached is a visio of the layout.  So basically 3 stacks terminate to a ms350. 2 of the stacks work fine, the stack where the access port to the 3rd party is doesn't work..which doesn't make sense.. to add to the mystery, i can take a laptop and plug into the stack and it works fine. but when i move a device over from old network equipment..nothing.  I'm stumped

[Nick]

So if am following this correctly you mean that stk2 and stk3 can route to VLAN 101 through the 800 okay but stk1 cannot?

[leadtheway]

correct, and stk1 is where the 800 is connected..lol

[PhilipDAth]

Anything interesting the switch event log?

Have you made the MS350 the root of your spanning tree?

Are you running 10.x firmware of better?

[Nick]

Hmm that is interesting. 

So if you connect to stk1 you receive DHCP / or you can ping the rest of the existing network?

Could it be that kit moved from the old network is still being remembered somehow? Have you tried introducing fresh connections to the stk1?

[leadtheway]

Yes the 350 is the RSTP root. Firmware is 10.45.

Caveat to the laptop working..i think its arp cache is what allow it to work.. If i try to do a release/renew while plugged into stk1 it can't find dhcp etc...but if using its cached address it can.., if I plug it into either of the other two stacks it works fine, release/renew works as intended.  Doing packet trace with meraki support they gave me this:

Dell workstation connected to sw3/stk1 / port 48 is sending ARP requests for its gateway, i.e. 10.209.152.254, but it is not receiving any ARP replies. We see these ARP requests going out of sw1/stk1 / port 1.



Furthermore, it looks like there may be some asymmetric routing going on upstream of sw1/stk1 / port 2, as:

- Outbound traffic, i.e. from laptop to Internet, is using an HSRP MAC address as the destination MAC

- Inbound traffic, i.e. from Internet to laptop, is using a Fortinet (90:6c:ac:3e:ec:d6) source MAC address

[Nick]

Hmm it starts to get sticky then....

So something is filtering or not replying to the DHCP requests. The VLAN's on stk1 switches work as you can set manually without an issue?

Can you get a look at whats going on upstream or is it all locked away?

[leadtheway]

yeah i can set the vlan on the stk no problem....the part that is crazy is, there are 3 stacks at this site all plugged into the 350, the stack that has the upstream router plugged into it is the one that isn't working

[LouisR]

If you were configure the 350 to access the Cisco 800 directly instead of going through STK1. Do STK2 and 3 still have no issues accessing the routes they need? Seems like there is a crazy hairpin going on in STK1. Is anything showing up in the event logs like BDU guards or root guards triggering?

[leadtheway]

even logs from which switch?

[leadtheway]

Just wanted to update everyone in case someone comes here with similar issue. So actually onsite there this weekend and got to troubleshoot with hands on.  It was the weirdest thing..I could run pcap and could see the arp going out of the device but not getting an answer, yet in the Cisco 800 i could see it updating its arp table yet not receiving any arp request. So when the device would even try to ping it wouldn't go anywhere.  But plugging in a device that has communicated before and was in the arp table it would work.  Craziest thing i've ever seen.  So resolution? Rebooted the 800 cisco...lol

[Nick]

And that resolved the issue?

[leadtheway]

Yes the 350 is the RSTP root. Firmware is 10.45.

Caveat to the laptop working..i think its arp cache is what allow it to work.. If i try to do a release/renew while plugged into stk1 it can't find dhcp etc...but if using its cached address it can.., if I plug it into either of the other two stacks it works fine, release/renew works as intended.  Doing packet trace with meraki support they gave me this:

Dell workstation connected to sw3/stk1 / port 48 is sending ARP requests for its gateway, i.e. 10.x.x.254, but it is not receiving any ARP replies. We see these ARP requests going out of sw1/stk1 / port 1.



Furthermore, it looks like there may be some asymmetric routing going on upstream of sw1/stk1 / port 2, as:

- Outbound traffic, i.e. from laptop to Internet, is using an HSRP MAC address as the destination MAC

- Inbound traffic, i.e. from Internet to laptop, is using a Fortinet (90:6c:ac:3e:ec:d6) source MAC address

[DeeWilliams]

Good day,

Im looking to create some network diagrams, i am using Meraki Switches. where can i find the icons you are using on your diagram

[leadtheway]

https://meraki.cisco.com/blog/2016/05/visio-stencils-now-available/