Malformed ISAKMP and L2TP across the Network

[MSakr]

Hi Gurus

As of 2 days ago, we started to see a lot of malformed packets on our network, inbound and outbound and on the internal appliances, these are coming from/to our Zerotrust provider

It happens only in one office location with a hundred employees.. any clues?

Meraki support are pointing to ISP, then we said why then these are coming from inside? they said to check the SASE provider.. clients have a SASE agent installed and they are not able to connect and work with frequent QOS and drops if they connect to Zerotrust..

SASE provider says it is the ISP.. ISP says nothing on their side.. 

We have 2 MX95s in HA and lots of MS and MRs on that Network, all Meraki architecture..

Thanks

[alemabrahao]

Have you tried deleting the existing connection and creating a new one?

Is it possible to test Anyconnect?

Have you had any recent changes to the network, such as a firmware update for example?

Have you opened a support case with Meraki for them to investigate to make sure it's not a bug?

[MSakr]

Hi

Yes, Meraki support actually pointed to the Malformed packets, we are on the latest firmware.. they mentioned that there is a bug on the latest with VPN.. might it be related? but we have been on that firmware for over a month now and issues started to happen only Monday

We use Cloudflare SASE, these guys are blaming the ISP.. the ISP says all good.. we are trying by elimination..

We have an edge MS stack segmented with a vlan exposed to the internet to accomodate the ISP 2 routers and BGP virtual IP.. this stack is part of the same network dashboard and the MX is connected to it on one hand.. I suspect it might be something to lok at as that Network is clocking 32K clients and these are all the public IPs seems that connected to that stack on the public VLAN..

[alemabrahao]

So you're talking about an S2S VPN tunnel with a non-Meraki peer?

If so, I would try another firmware version to test this.

[MSakr]

No, it is not a S2S, it is a Zerotrust/SASE solution more P2S

But we see traffic degradation also on non connected devices

[MSakr]

And also we started to get Disabled Gateway (Bad DNS) errors on the MX uplink randomly

[MSakr]

So update.. Meraki support didn;t much help rather pushing it on ISP or Cloudflare..

We disabled a S2S vpn that was set with a cloud provider and the issue seems to be solved.. now it might be related to the latest MX bug with VPN.. we need to make sure our VPN can be brought up too..

[MSakr]

Upd: patched, had to go to the schedule upgrade to see that the minor version was available

Now trying to patch the MX to the latest release, however somehow it is greyed out and we are off by .2 versions and the .2 is a stable release not beta

I see this

FIRMWARE

Up to date

Current version: MX 18.211

[MSakr]

Upd: Unfortunately the issue persists.. there was a case related to L7 blocked p2p being falsly identified as cloudflare warp traffic, but in our case we don;t block that traffic.. back to square 1 and Meraki support is to no avail.. thinking of opening a new tocket to get another support on the call and see how this would go..

Nothing in the MX logs suspicious..

[MSakr]

Now this sounds more crazy.. with the above packets we had disruptions in traffic.. we were able to reproduce the problem by loading the uplink.. any upload would trigger traffic disruptions from Zerotrust connected clients.. looks like shaping is applied upstream..

[MSakr]

Hi All

The root cause of the issues were a conjunction of issues related to unreachable destinations from the ISP provider to the SASE provider, while all other traffic was ok, this led to the SASE clients to send tens of retries spiking the connections limits on the Firewalls and bringing everything down.. 

Unfortunately the MX firewalls have no mean to detect such spikes in connections, we had these highlighted on Firepower firewalls.. hopefully the MEraki team will implement such details along with much needed CPU, memory and storage usage on the devices..