Strange VLAN issue today

[OVERKILL]

One of my customers has an MX84 running 14.53 stable. This AM a Cisco Small Business switch connected to one of the HPE switches that are the main access switches for their the LAN stopped passing traffic on the primary (untagged) VLAN. All devices attached directly to the HP switches were fine. 

Figuring the issue was with the Small Business switch the service provider for the software solution that the SB switch was associated with had them reboot the switch. This made no difference.

I grabbed a clean 2960S and dragged that in to troubleshoot.

- Plugged it into one of the HP's, it wouldn't pass traffic on the primary VLAN. 

- Plugged it directly into the MX84, it wouldn't pass traffic on the primary VLAN.

- Set the port I was plugged into as an access port for VLAN 2? Passed traffic no problem. 

I ended up rebooting the MX84 and then everything just worked properly again. The entire time, anything plugged into the HP switches were fine and VLAN 2 was wholly unaffected. Quite odd! 

[Inderdeep]

This is something unexpected and really odd. You resolved your problem with rebooting MX84 ? 

[OVERKILL]

Yes, that's all it took, just restarting the MX84. 

[Inderdeep]

I am not sure may be TCAM table is full on the device leads to this kind of issue. I am glad you restored back ! Happy long weekend !

[cmr]

@OVERKILL I don't know if it would have caused you to not experience the issue but development has stopped on the 14.x train.  There is a 14.55 version but it is only applied if you have an old MX on a network assigned to the 15.x train.

15.x is at 15.42 and is release candidate level and stable (we use it on over 20 MXs)

16.x is at 16.4 and is beta level and a number of people report it works okay for them so far, the main addition is AnyConnect client VPN support

17.x is in the early stages of a closed beta and is progressing, the main addition here is IPv6

In summary, I would strongly suggest upgrading to 15.42 at the moment, there are a lot of improvements over 14.x

[OVERKILL]

Yeah, I was on the on AnyConnect beta, so I'm familiar with the 16.x series, it's just amusing that the "stable" train had this issue, as it should be the most mature code. 

I have two MX84's on 16.4 now. Had an interesting issue with cloud auth not working for a few hours (that self-rectified) on 16.3 at one site. Never experienced it elsewhere. 

Bumping it to the 15.x series definitely sounds like a worthwhile endeavour. I'm going to take the opportunity to swap out the HP switches for a stack of 2960 ones, so will likely do the code bump at the same time. 

[PhilipDAth]

Tips:

* Use a single connection to the MX84 to prevent spanning-tree loops.

* Configure the HP switch to use MSTP instead of its variant of RSTP.

* Upgrade the firmware on the small business switch.

* If the small business switch supports it, change it to run MSTP.

I don't know about your specific models, but Cisco and HP don't always use the same weights for ports when they run RSTP.  This can cause spanning-tree inconsistencies.  It can be triggered by simply bring ports up and down in a specific order.

For MSTP they do.

I have had this exact issue in the past in mixed Cisco and HP switch environments - and changing everything to use MSTP has solved it every time.

[OVERKILL]

There was indeed just a single uplink to the MX84 at the time of the issue. I connected the 2960 to the MX84 on a separate port just to test, it made no difference. It would pass VLAN2 traffic on the switches that it wouldn't pass VLAN1 traffic on, which was the most interesting bit. 

The MX84 has been running without issue for about a year. 

I'm pulling the Cisco SB switch as they are changing software vendors. I'm putting in a stack of 2960's for the main LAN, VLAN2 will most likely stay on an HP but we'll see what port utilization looks like after the 3x 2960's are installed. 

All the HPE switches are using 802.1D, not RSTP (802.1W).

I unfortunately don't have access to the configuration on the Cisco SB switch, since it was vendor provided. That's another reason I'm inclined to punt it once they are out of the picture.