Malformed ISAKMP and L2TP across the Network

MSakr
Getting noticed

Malformed ISAKMP and L2TP across the Network

Hi Gurus

As of 2 days ago, we started to see a lot of malformed packets on our network, inbound and outbound and on the internal appliances, these are coming from/to our Zerotrust provider

 

It happens only in one office location with a hundred employees.. any clues?

Meraki support are pointing to ISP, then we said why then these are coming from inside? they said to check the SASE provider.. clients have a SASE agent installed and they are not able to connect and work with frequent QOS and drops if they connect to Zerotrust..

 

SASE provider says it is the ISP.. ISP says nothing on their side.. 

We have 2 MX95s in HA and lots of MS and MRs on that Network, all Meraki architecture..

 

ThanksScreenshot 2024-06-12 at 10.19.44.png

 

 

10 Replies 10
alemabrahao
Kind of a big deal
Kind of a big deal

Have you tried deleting the existing connection and creating a new one?
 
Is it possible to test Anyconnect?
 
Have you had any recent changes to the network, such as a firmware update for example?
 
Have you opened a support case with Meraki for them to investigate to make sure it's not a bug?
I am not a Cisco Meraki employee. My suggestions are based on documentation of Meraki best practices and day-to-day experience.

Please, if this post was useful, leave your kudos and mark it as solved.
MSakr
Getting noticed

Hi

 

Yes, Meraki support actually pointed to the Malformed packets, we are on the latest firmware.. they mentioned that there is a bug on the latest with VPN.. might it be related? but we have been on that firmware for over a month now and issues started to happen only Monday

We use Cloudflare SASE, these guys are blaming the ISP.. the ISP says all good.. we are trying by elimination..

We have an edge MS stack segmented with a vlan exposed to the internet to accomodate the ISP 2 routers and BGP virtual IP.. this stack is part of the same network dashboard and the MX is connected to it on one hand.. I suspect it might be something to lok at as that Network is clocking 32K clients and these are all the public IPs seems that connected to that stack on the public VLAN..

 

alemabrahao
Kind of a big deal
Kind of a big deal

So you're talking about an S2S VPN tunnel with a non-Meraki peer?
 
If so, I would try another firmware version to test this.
I am not a Cisco Meraki employee. My suggestions are based on documentation of Meraki best practices and day-to-day experience.

Please, if this post was useful, leave your kudos and mark it as solved.
MSakr
Getting noticed

No, it is not a S2S, it is a Zerotrust/SASE solution more P2S

But we see traffic degradation also on non connected devices

MSakr
Getting noticed

And also we started to get Disabled Gateway (Bad DNS) errors on the MX uplink randomly

MSakr_0-1718204508700.png

 

MSakr
Getting noticed

So update.. Meraki support didn;t much help rather pushing it on ISP or Cloudflare..

We disabled a S2S vpn that was set with a cloud provider and the issue seems to be solved.. now it might be related to the latest MX bug with VPN.. we need to make sure our VPN can be brought up too..

MSakr
Getting noticed

Upd: patched, had to go to the schedule upgrade to see that the minor version was available

Now trying to patch the MX to the latest release, however somehow it is greyed out and we are off by .2 versions and the .2 is a stable release not beta

 

I see this

 

 

FIRMWARE
Up to date
Current version: MX 18.211

 

MSakr
Getting noticed

Upd: Unfortunately the issue persists.. there was a case related to L7 blocked p2p being falsly identified as cloudflare warp traffic, but in our case we don;t block that traffic.. back to square 1 and Meraki support is to no avail.. thinking of opening a new tocket to get another support on the call and see how this would go..

Nothing in the MX logs suspicious..

MSakr
Getting noticed

Now this sounds more crazy.. with the above packets we had disruptions in traffic.. we were able to reproduce the problem by loading the uplink.. any upload would trigger traffic disruptions from Zerotrust connected clients.. looks like shaping is applied upstream..

MSakr
Getting noticed

Hi All

The root cause of the issues were a conjunction of issues related to unreachable destinations from the ISP provider to the SASE provider, while all other traffic was ok, this led to the SASE clients to send tens of retries spiking the connections limits on the Firewalls and bringing everything down.. 

 

Unfortunately the MX firewalls have no mean to detect such spikes in connections, we had these highlighted on Firepower firewalls.. hopefully the MEraki team will implement such details along with much needed CPU, memory and storage usage on the devices..

Get notified when there are additional replies to this discussion.
Welcome to the Meraki Community!
To start contributing, simply sign in with your Cisco account. If you don't yet have a Cisco account, you can sign up.
Labels