- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Malformed ISAKMP and L2TP across the Network
Hi Gurus
As of 2 days ago, we started to see a lot of malformed packets on our network, inbound and outbound and on the internal appliances, these are coming from/to our Zerotrust provider
It happens only in one office location with a hundred employees.. any clues?
Meraki support are pointing to ISP, then we said why then these are coming from inside? they said to check the SASE provider.. clients have a SASE agent installed and they are not able to connect and work with frequent QOS and drops if they connect to Zerotrust..
SASE provider says it is the ISP.. ISP says nothing on their side..
We have 2 MX95s in HA and lots of MS and MRs on that Network, all Meraki architecture..
Thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Please, if this post was useful, leave your kudos and mark it as solved.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi
Yes, Meraki support actually pointed to the Malformed packets, we are on the latest firmware.. they mentioned that there is a bug on the latest with VPN.. might it be related? but we have been on that firmware for over a month now and issues started to happen only Monday
We use Cloudflare SASE, these guys are blaming the ISP.. the ISP says all good.. we are trying by elimination..
We have an edge MS stack segmented with a vlan exposed to the internet to accomodate the ISP 2 routers and BGP virtual IP.. this stack is part of the same network dashboard and the MX is connected to it on one hand.. I suspect it might be something to lok at as that Network is clocking 32K clients and these are all the public IPs seems that connected to that stack on the public VLAN..
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Please, if this post was useful, leave your kudos and mark it as solved.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
No, it is not a S2S, it is a Zerotrust/SASE solution more P2S
But we see traffic degradation also on non connected devices
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
And also we started to get Disabled Gateway (Bad DNS) errors on the MX uplink randomly
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
So update.. Meraki support didn;t much help rather pushing it on ISP or Cloudflare..
We disabled a S2S vpn that was set with a cloud provider and the issue seems to be solved.. now it might be related to the latest MX bug with VPN.. we need to make sure our VPN can be brought up too..
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Upd: patched, had to go to the schedule upgrade to see that the minor version was available
Now trying to patch the MX to the latest release, however somehow it is greyed out and we are off by .2 versions and the .2 is a stable release not beta
I see this
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Upd: Unfortunately the issue persists.. there was a case related to L7 blocked p2p being falsly identified as cloudflare warp traffic, but in our case we don;t block that traffic.. back to square 1 and Meraki support is to no avail.. thinking of opening a new tocket to get another support on the call and see how this would go..
Nothing in the MX logs suspicious..
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Now this sounds more crazy.. with the above packets we had disruptions in traffic.. we were able to reproduce the problem by loading the uplink.. any upload would trigger traffic disruptions from Zerotrust connected clients.. looks like shaping is applied upstream..
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi All
The root cause of the issues were a conjunction of issues related to unreachable destinations from the ISP provider to the SASE provider, while all other traffic was ok, this led to the SASE clients to send tens of retries spiking the connections limits on the Firewalls and bringing everything down..
Unfortunately the MX firewalls have no mean to detect such spikes in connections, we had these highlighted on Firepower firewalls.. hopefully the MEraki team will implement such details along with much needed CPU, memory and storage usage on the devices..
