Massive amounts of VRRP transitions

[theshmike]

Hi,

due to the event log in one of our networks, we've been experiencing massive amounts of VRRP transitions for weeks between two of our MS425-32.

One some days, the log shows up 100+ transitions, even multiple transitions in the same minute.

The support only told us that there seems to be a connectivity issue and such things. We've tested multiple paths between the switches, but the transitions didn't stop.

There are only two more devices in the network that are doing VRRP. They are operating on different VRIDs (shame on Meraki that one could not set the VRID for VRRP btw) so there is no VRID colission. I also did lots of packet capturing to make sure that there are no unknown devices that are doing VRRP.

Strangely, we did not experience any connectivity/routing issues during the times that the log shows the transitions.

Does anybody have an idea how to further investigate into this or solve this?

[jdsilva]

---

@theshmike wrote:

(shame on Meraki that one could not set the VRID for VRRP btw)

---

They also don't use a standard multicast MAC address... So even if you could set VRID it still wouldn't work right. And they also put every VLAN onto one virtual route which is a little off in my experience, but within RFC.

I'd probably approach this by starting with packet captures to see if the advertisements are truly lost. If they are then work through the path to see if you can find where. 

[theshmike]

---

@jdsilva wrote:

---

I'd probably approach this by starting with packet captures to see if the advertisements are truly lost. If they are then work through the path to see if you can find where. 

---

The switches have a direct connection and I already exchanged the SFP+ modules to make sure that they are not faulty.

What would you suggest to do check if advertisements are lost? Place a wireshark on each side and do a long-term-compare on the advertisements sent on one and received on the other side?

Very annoying job, as the times on which the transitions are happening are not predictible 😞

[jdsilva]

You could try a mirror port to a laptop with Wireshark, yup. Or use the dashboard packet capture. In a perfect world you'd want to capture both sides so you can compare the two. Given the switches are directly connected you'd sure expect they to be the same...

Since Meraki MS VRRP sends hellos every 0.3s, with a dead timer of 0.9s, you may actually be able to see missed hellos happening even though a VRRP transition doesn't occur. If only one or two hellos are missed it wouldn't be enough to trigger a transition, but you would be able to see they are missed in the capture. 

[theshmike]

---

@jdsilva wrote:

Since Meraki MS VRRP sends hellos every 0.3s, with a dead timer of 0.9s

That is weird, because if it is like this, I am wondering how two transitions can happen in the same second:

---

Jan 2 19:40:06	VRRP transition	"master>backup"
Jan 2 19:40:06	VRRP transition	"backup>master"
Jan 2 19:40:04	VRRP transition	"master>backup"
Jan 2 19:40:04	VRRP transition	"backup>master"
Jan 2 19:40:02	VRRP transition	"master>backup"
Jan 2 19:40:01	VRRP transition	"backup>master"
Jan 2 19:40:00	VRRP transition	"master>backup"
Jan 2 19:40:00	VRRP transition	"backup>master"

Or does it maybe failover after 0.9 and preempt back after 0.1?

[PhilipDAth]

If the issue wasn't happening before then it could also be a software bug on the MS affecting one or both MS.  I'm going to guess this wont be easy but any chance to schedule in a reboot (and perhaps bring them up to the current stable firmware at the same time)?

[theshmike]

---

@PhilipDAth wrote:

this wont be easy but any chance to schedule in a reboot (and perhaps bring them up to the current stable firmware at the same time)?

Yes, that's currently the last option on our action list. There's nothing about such an issue in the release notes, but you never know...

---