MS250 stack, dhcp relay

[bmarms]

I have a Meraki case opened but wanted to post here as well.  I have an MS250 stack of two switches running 11.31 code.  The switch stack is configured with several vlans and vlan interfaces all setup to relay dhcp to offsite dhcp servers.  i have routers and firewalls connected to each switch in the stack for full diversity.  switch 2 is the master based on lowest mac address.  

routing is a default route from the switch stack to the fw-lan side ip.  fw-wan side has a default route to the routers.  both firewalls and routers are active/passive HA

physical connectivity is:

fw1-lan to sw1 on L3 vlan 1, fw1-wan and rtr1-lan to sw1 on L2 vlan 1060

fw2-lan to sw2 on L3 vlan 1, fw2-wan and rtr2-lan to sw2 on L2 vlan 1060  

clients requesting a dhcp address works fine via both firewalls and rtr2 connected to sw2

clients requesting a dhcp address does not work via both firewalls and rtr1 connected to sw1

process of elimination troubleshooting has revealed the following:

If both routers are connected to sw2, dhcp works fine through both of them

if both routers are connected to sw1, dhcp does not work via either router

if the routers are connected directly to the firewalls (bypassing the switches on the WAN side), dhcp works via both routers

if i power off sw2, making sw1 the master, dhcp works via rtr1 connected to sw1

once sw2 was powered back up and returned to the stack, dhcp continued to work via rtr1 but now did not work via rtr2.  assuming this is due to sw1 now having the master role?

a reboot of sw1 returned the master role back to sw2 and I am back to where i began.  dhcp works via rtr2 connected to sw2 but not rtr1 connected to sw1

my meraki engineer has me upgrade to 12.28.  once my stack was on that version anything connected to sw1 was dropping 50pct or more of their packets.  devices connected to sw2 were fine.  we rolled back to 11.31.

anyone else having any similar issues?  i've supplied meraki with pcaps.  i see the dhcpoffer returning from our dhcp server via both router interfaces.  i am not seeing it on the pcaps on my firewall interfaces which tells me the switches are dropping it for some reason even though it is working via sw2.  the src and dest mac on the dhcpoffer packet have been confirmed to be a src of my router mac and dest of my firewall mac.

looking for any additional help/advice

[PhilipDAth]

It sounds like you have a loop somewhere.

These firewalls.  Do they have a single connection to the switches each, or two connections (I'm going to guess two connections ...).

[bmarms]

2 connections each.  one on the LAN side and one on the WAN side.  the interfaces are routed, not switched, so there are no BPDU's, etc.  STP is also enabled on the meraki ports connected to the firewalls

FWIW, Meraki support has confirmed it's an issue with the non-master switch role in the stack and indicated code version 12.28 should fix it but it broke everything on the non master switch on my MS250 stack.  they're looking into that issue now in development

[cmr]

@bmarms 12.30 had some more fixes and now 14.5 is out, I'd try that at a quiet time (and roll back if it is an issue)

Failing that, can you not stack the switches, I always find that works better than warm spare (from Cisco IOS days when we used to do that)

[bmarms]

shouldve updated this one.  there's some bug in their code.  the fix was to add another L3 vlan interface, confirm the switches said "up to date" in the dashboard, then i could remove the new interface.  something to do with the route table on both switches in the stack weren't synced.