MX450 Packet Loss

[mattberk]

We recently replaced MX250 with MX450 in HA pair. We have these firewalls in front of a service that pulls around 3-4 Gbps. The MX250's were constantly failing over at peak traffic times, so we replaced them with 450s. The 450 ran okay the first day, but now are showing packet loss. We are uplinked in a data center 10 Gbps with DAC. 

We are running the 19.1.8 RC firmware. Any issues with this firmware? Should we go down to stable release? The firewall is not failing over, when it primary becomes unresponsive, the dashboard drops out temporarily, and latency is high.

Any insight or ideas would be appreciated.

[rhbirkelund]

Does the packet loss seem to be to and from Internet? Do you notice the same packet loss on inter-VLAN traffic?I'm wondering if the cause of packet loss may be due to further upstream, i.e. ISP. 

You mention you're using DAC cables, have you considered replacing the DAC cable with Meraki SFPs?

[mattberk]

Thank you for responding. I took over this setup so I am still learning it. I went out to the data center. We actually have Cisco C9300NM-8X switches that the MXs are connected to via DAC. Then those switches have the uplink to the data center cross connects via SFP and fiber.

ISP has reviewed and stated no issues on their end. I will have to try and figure out how to determine if we are seeing packet loss between the switches when the traffic occurs.

[PhilipDAth]

Do those NM-8X interfaces show any packet loss?

[Blue_Bird]

The packet loss issue with your Meraki MX450s, especially after an upgrade and with high traffic, could be due to a few factors, including potential firmware bugs in 19.1.8 RC, resource limitations, or network configuration issues. It's recommended to first verify network performance and then consider downgrading to a stable release, potentially 19.1.7

https://documentation.meraki.com/General_Administration/Firmware_Upgrades/Meraki_Firmware_Release_Pr...

[CarolineS]

Hi @Gopinath_Pigili - Was this response generated by AI? If so, please cite your sources, per community guidelines.

[cmr]

How much bandwidth is in use when they fail?

Are they in routed mode, or other?

What license do they have and what features are configured?

How many users and devices are going through them?

[mattberk]

It peaks at around 4.1 Gbps but averages at about 2.8-3.2 Gbps.

Routed mode, they are connected via DAC to Cisco C9300NM-8X switches. Then those switches have the uplink to the data center cross connects via SFP and fiber.

Advanced security licenses. We have all the security features turned off. The services they are hitting are 1:1 NAT rules.

It is a service, so we have our customers uploading data with about 400-700 jobs at a time between everyone.

[RaphaelL]

What is the current device utilization ? Does it peak 100%?

[mattberk]

Seems to average around 75%. This was from last night. The previous MX250's would peak 100% and then fail over.

[RaphaelL]

That's interesting. Above 70% I usually experience small loss or a slight increase in latency on my devices.

Do you have IDS/IPS enabled ?

[mattberk]

IDS/IPS is disabled. Running in routed mode.

[RaphaelL]

I would suggest to ask support for the detailed stats. There is more than device utilization , they can see PPS , flows , memory and more.

Does the loss only occur at specific moments ? if so I'm sure it would match one of the detailed stats peak.

[cmr]

I'd definitely ask for this as the graph is averaged over a time period, so 70% could well have peaks that are higher.  I'd speak to the supplier as there is an MX650 now and it's possible even the MX450s are undersized...

[RWelch]

If it were me, I'd be inclined to initially swap out the DAC with SFP+ modules and fiber to rule out possible upstream issues with the ISP as indicated by @rhbirkelund. If that doesn't bring any clarity then look at changing to a different firmware (rollback).  Was the same DAC cable used with the MX250s?  Your post doesn't elaborate on where exactly you are seeing packet loss.

Is this perhaps a MTU issue?
Troubleshooting MTU Issues 

If this is a production network, it might be a good idea to submit a support ticket to leverage supports assistance with what they might be able to share from the backend.

[mattberk]

Thank you for responding. I took over this setup so I am still learning it. I went out to the data center. We actually have Cisco C9300NM-8X switches that the MXs are connected to via DAC. Then those switches have the uplink to the data center cross connects via SFP and fiber.

ISP has reviewed and stated no issues on their end. I will have to try and figure out how to determine if we are seeing packet loss between the switches when the traffic occurs.

I tried to open a ticket last night and they were not able to help out. I will need to try and call back in today.

I will review the MTU guide and see how we have that configured.

[RWelch]

It might be worth making sure the DAC cables are "seated" sufficiently since it's a setup you are getting familar with versus trusting what someone else might have done.  Possible that one or both ends aren't seated fully - as a suggestion.

Troubleshooting Packet Loss between Devices 

[cmr]

Are you using Cisco / Meraki DAC cables?

[mattberk]

The cable between the MXs and switches are CBL-TA-1M-AO

[cmr]

They should be fine then 🤔

[mattberk]

Thank you for the insight. On the phone with Meraki support. We were seeing client IP conflict on uplink errors that relates the virtual WAN mac address potentially. 

Jobs are running fine right now on our spare, wondering if there is an issue with the HA setup or the primary MX.