Multiple Services Impacted when Pushing Config

DazKew
Here to help

Multiple Services Impacted when Pushing Config

Hi all, 

We're in the middle of a major migration from WatchGuard to 4 x MX250s, 1 x MX67, 75 x Z3s. Whenever we push out any configuration changes to firewall rules or static routes, multiple systems such as Twilio, Webex and AnyConnect (coming off a connected FTD) are failing temporarily and we are seeing RST packets coming back from them in PCAPs. 

We've had a few issues with the support engineer saying that this is the cloud providers fault, and basically saying telling us this is normal behaviour and won't be fixed -- not a great resolution really! Currently trying to escalate this as far as possible. 

Obviously, this wasn't happening on the WG appliances we pulled out, so have a customer (quite rightly) looking for Meraki to take some ownership of this. Looking at all release notes for current and future MX versions, we can see this:

Known issues

  • After making some configuration changes on MX84 appliances, a brief period of packet loss may occur. This will affect all MX84 appliances on all MX firmware versions.

I am starting to think this may be a wider issue than the MX84 as we have proven issues on both the 250 and Z3. Has anybody else noticed this issue? Can anyone from Meraki Dev provide any insight? Currently we can only make changes to the MXs in the early hours. 

3 REPLIES 3
PhilipDAth
Kind of a big deal
Kind of a big deal

That issue mentioned in the firmware release only affected the MX84's (which have now had EOS announced).  It was mostly around changing changes to the interfaces (such as changing VLANS, VLAN trunking, etc).

 

I haven't worked on any MX250's, but none of the other products (including the MX84) looses any packets during a firewall rule change.  I do firewall rule changes a lot.

 

You can't argue with a packet capture.  Perhaps the issue is specific to the MX250, or perhaps it is related to the firmware version being used on the MX250.  Are you able to upgrade the firmware on just one of the MX250's and do some further experiments?

 

ps. I've found AnyConnect on MX to be very solid.

Thanks for the reply, Philip. 

We are no rookies to the MX ourselves, as an MSSP we support hundreds of MX and Z3 devices and I agree this is not usual behaviour. On this occasion I think I can argue against the packet capture as pushing a new rule should not impacts seemingly all external services. This is why I am so perplexed at Meraki's advice that this is normal behaviour. 

 

The AnyConnect here isn't on the MX, but on a Firepower which connects back through the MX. 

99% of the time, we can get around the limitations in the MX (with things like SNAT), but we seem to be hitting a brick wall with this one and Meraki support won't budge. You should be able to push firewall changes in the middle of the day without impacting an entire business - surely! 

Is your topology map also screwy and clients showing up on switches which they are not even physically connected.   I’m about to give up on Meraki unless they start changing their act.   My lease is up in 25 days guess will be going either back to WatchGuard or Untangle.

Get notified when there are additional replies to this discussion.
Welcome to the Meraki Community!
To start contributing, simply sign in with your Cisco account. If you don't yet have a Cisco account, you can sign up.
Labels