Avoid 15.44 if you are running a warm spare configuration

PhilipDAth
Kind of a big deal

Avoid 15.44 if you are running a warm spare configuration

I've been asked to investigate increasing numbers of issues with customers running 15.44 warm spare configurations.
 
What I have found is MXs in this specific configuration seem to experience regular crashes that are not logged.  Only Meraki support can see the crashes and reboots.
 
You can spot it by going to the event log and filtering on VRRP transitions.  You'll find customers go from have none or very few transitions, to having them more regularly on 15.44.
If the MXs plug into a Meraki switch you see logged events where a port goes down for a minute and a half or so, and then comes up (during the reboot).
 
Common symptoms that customers will report is their Window's VPN client connections drop randomly.  It affects everyone's connections at the same time.
 
I have had one extreme customer with the issue which required their MXs to be hard power cycled on each crash, which for them happen every 12 to 24 hours.  This is an extreme case.  All the others I have investigated simply crash and reboot without needing human intervention.
 
 
I've been advising customers to move to 16.12 if they are affected.  16.12 works really well.
 
Remember, only warm spare configurations appear to be affected.  Standalone MXs seem to be fine.
15 REPLIES 15
ww
Kind of a big deal
Kind of a big deal

Re: Avoid 15.44 if you are running a warm spare configuration

All models?

Both routed and concentrator mode?

PhilipDAth
Kind of a big deal

Re: Avoid 15.44 if you are running a warm spare configuration

I've noticed it mostly in lower-end models like MX68's and MX84's - but there are simply more of these.

 

Only had to investigate routed mode configurations so far.

UCcert
Kind of a big deal

Re: Avoid 15.44 if you are running a warm spare configuration

Hi @PhilipDAth , yep, this was the same issue that a friend of mine ran into a couple of weeks back and was getting the run around.

 

They were running 15.44 code, 15.42.3 and 15.42 all with issues namely their auto VPNs dropping every few hours.

Darren O'Connor | uccert.co.uk
https://www.linkedin.com/in/darrenoconnor/

I'm not an employee of Cisco/Meraki. My posts are based on Meraki best practice and what has worked for me in the field.
Inderdeep
Kind of a big deal

Re: Avoid 15.44 if you are running a warm spare configuration

@PhilipDAth : Good information sir 

Regards
Inderdeep Singh
www.thenetworkdna.com ( Awarded by Cisco IT Blogs award 2020)
Aaron_Wilson
A model citizen

Re: Avoid 15.44 if you are running a warm spare configuration

I have a mx100 HA pair that is not showing anything in logs with vrrp. Just registry up/down.

 

I have two mx400 HA pairs going to 15.44 this week and next......🤔

PhilipDAth
Kind of a big deal

Re: Avoid 15.44 if you are running a warm spare configuration

>I have two mx400 HA pairs going to 15.44 this week and next

 

Let me know how the MX400's go ... I have another customer with a lot of sites hanging off it, and I'm very nervous about letting their upgrade proceed.

Their MX400's are used exclusively for AutoVPN.  The others I have been investigating so far all use routed mode and do have Internet traffic flowing through them.

PhilipDAth
Kind of a big deal

Re: Avoid 15.44 if you are running a warm spare configuration

>I have a mx100 HA pair that is not showing anything in logs with vrrp. 

 

@Aaron_Wilson , are they being used for Internet access as well, or only AutoVPN?

Aaron_Wilson
A model citizen

Re: Avoid 15.44 if you are running a warm spare configuration

Autovpn comes in over internet (wan). But any traffic destine for the internet heads south, not hairpin.

Aaron_Wilson
A model citizen

Re: Avoid 15.44 if you are running a warm spare configuration

@PhilipDAth- had a MX400 HA pair upgrade last night. I'll just explain the deployment:

 

Old firmware was a 14.x flavor

 

This is a DC head-end. Auto-vpn comes in from the north (internet), all internet and non-Meraki traffic heads south into the DC core. All Meraki destine traffic heads back north to other hubs/spokes.

 

Warm spare, however it is an east/west direct connection (yea yea, I know).

 

It's been almost 12 hours and the only VRRP transition was during the upgrade. There were some ethernet port carrier logs about 10 minutes later, but it was only a few logged on the primary and have since subsided.

 

Here is the logs for the location. I filtered on VRRP and port given all the other boring registry stuff that gets logged.

 

Aaron_Wilson_0-1633087649304.png

 

jbright
Getting noticed

Re: Avoid 15.44 if you are running a warm spare configuration

I have a pair of MX450 running 15.44 and I do not see this problem.

TEAM-ind
Getting noticed

Re: Avoid 15.44 if you are running a warm spare configuration

Please post follow up if/when you proceed with 15.44 on the mx400s.   I've been holding off on my mx400 HA pair upgrade, due mostly to procrastination.  But seeing this discussion, I am glad I procrastinated.

Aaron_Wilson
A model citizen

Re: Avoid 15.44 if you are running a warm spare configuration

@TEAM-indI did go to 15.44 on one set of MX400, so far so good.

 

I have another set going in a week.

 

Aaron_Wilson
A model citizen

Re: Avoid 15.44 if you are running a warm spare configuration

2nd set of MX400 warm spares moved to 15.44. Problem free!

Gillic01
Conversationalist

Re: Avoid 15.44 if you are running a warm spare configuration

Our company has had issue with 15.44 where two separate mx 450 just stopped responding and had to be powercycled. Happened once then again 30 days later.

akh223
Getting noticed

Re: Avoid 15.44 if you are running a warm spare configuration

Looks like we got hit with this today.  Our HA pair of MX450's that act as our SD-WAN concentrators started having problems.  The primary went completely unresponsive and had to be power cycled to come back online.

The spare took over, and for some reason reported that it had a bad power supply, which it didnt....lights were fine.

 

Looks like I am going back to 15.43.

Get notified when there are additional replies to this discussion.
Welcome to the Meraki Community!
To start contributing, simply sign in with your Cisco account. If you don't yet have a Cisco account, you can sign up.
Labels