Meraki IDS Alerts across multiple MX's

[JessIT1]

Getting IDS blocked alerts for FILE-OTHER Kaspersky antivirus library heap buffer overflow - without optional fields

Virus Total shows no detections

One example of Source:

https://whois.domaintools.com/8.251.140.126

OrgName:        Level 3 Parent, LLC

[Jacobim]

We are seeing the same alerts on our end. Some source IPs belong to Level 3 as well.

[JessIT1]

Seeing a lot of these too - 

vip0x008.map2.ssl.hwcdn.net

209.197.3.8:80

[Jacobim]

We see similar source addresses (starting with vip). Are you by chance in Canada using Rogers as ISP?

[JessIT1]

No, these alerts are mainly coming from our Columbia, Missouri office MX

[JessIT1]

Also seeing a lot of these - a23-210-14-33.deploy.static.akamaitechnologies.com

[etb]

I'm seeing a ton of these IDS rule hits as well.  Rule ID 1-16295 "Kaspersky antivirus library heap buffer overflow - without optional fields".  

Over 6,100 starting around 4:45pm EST on Tues 6/13 (nothing noteworthy before then).  

I'm guessing that all of the remote sources are CDN's.  HWCDN, LLNW, Akamai, Edgecast, and Level 3.  4 of the top 10 have contributed to between 600 and 1000 events each, and others of the top 10 are 80 or fewer events each.

We are running firmware MX 18.107, and that has not changed recently.  I'm not aware of any other changes in our environment which could be contributing to this. 

I'm taking a wild guess that maybe something changed with NBAR definitions, and it's causing false-positives?  That said, I have no idea what legitimate traffic would be hammering from CDN's what looks like roughly every 4 to 5 minutes per each affected local computer at literally all hours of the day.

[JessIT1]

6-14-2023 - 1,008 alerts - 6-15-2023 - 1,017 alerts.

The MX having this issue just updated a few days ago to firmware MX 18.107. Before that update this issue did not exist.

So far this morning, no new alerts, however the 2 destination endpoints that were affected are not online yet..so we'll see.

[NickHurleyJP]

I can confirm that of 12 sites I have seen this anomaly, 1 "spiked" on Tuesday 6th, the next site was Friday 9th, the other 10 all on Wednesday the 14th. Hope this isn't another "microsoft traffic = BAD" situation. These sites do use Microsoft cloud services of one form or another.

[JessIT1]

I had opened a case with Meraki support - here are their 2 responses so far. I replied to them that others are having same issue and to escalate case with support..I'll report back what they say.

What you are seeing here are the reported security events by IDS/IPS and AMP. The events that you referenced have been blocked by IPS from the network. Please read here for more information about the security center events: https://documentation.meraki.com/MX/Monitoring_and_Reporting/Security_Center

The IDS/IPS SNORT algorithms takes care of identifying if a traffic pattern is potentially malicious, and it seems that it labeled that traffic as such. If you believe it is safe, you can feel free to whitelist them and they will then be allowed. 

[JessIT1]

Just got response from support - Absolutely, we can take a look to see if these events you are seeing are false positives. I will investigate further from my end and let you know my findings.

[AlexP]

Hey @etb 

NBAR is not used with Snort at all - the definitions come from what we load from TALOS.

If folks are seeing an upswing in this traffic, there's really not a lot our support team can say beyond "the MX received traffic that matched the signature in question"

For what it's worth, the signature in question does correlate with a very old CVE, so I don't believe it should provoke any concerns beyond simple noise: https://nvd.nist.gov/vuln/detail/CVE-2005-3142

[etb]

Thanks, @AlexP.  Understood regarding NBAR vs Snort.

It's a little tough to assume that 5+ different CDN's all started doing something different at the same time though.  (It wasn't terribly unusual for us to go a whole 2 weeks with no IDS events, but all of a sudden now we have thousands per day from multiple CDN's.)  Or similarly, I really wouldn't know I guess, but it seems unlikely that some application/service which we use leverages 5+ different CDN's such that that application/service would be generating this traffic from multiple CDN's.  Not impossible, but it seems more likely something that changed on the Meraki side.

I can try to dig into it a little more.  Should I be able to see this traffic in a PCAP on my WAN interface, or does IDS block traffic before it would even get captured?  My dashboard doesn't seem to offer the ability to inspect packets for this rule.

[El-Bandito]

Can confirm our organization is getting this as well.  Got a bunch of alerts for that buffer overflow relating to AKAMAI Technologies. 

Also getting these other ones that look like this, but not as many:  vip0x008.map2.ssl.hwcdn.net 

[Dunky]

We've started getting loads of these too:

We are running 18.107.2 and the alerts started on 13th June which by coincidence(?) is when we started rolling out 18.107.2 (from 17.10.n).

[JessIT1]

Email response from Meraki support this morning:  After taking a further look into the IDS/IPS alert seen, we do not believe it to be a false positive as the reason it was flagged is because it matched a signature.

If you believe this traffic to be safe, you can feel free to whitelist it by clicking the signature and turning whitelist to ON. 

[NickHurleyJP]

It's not a false positive..."it matches a signature". "Whitelist if you believe it's safe". It's not the alert existing that is the problem. Granted I have not seen the surge since but this response from Meraki seems less than helpful. 

[El-Bandito]

I mean, I highly doubt any of us are using Russian anti-virus software in our organizations, so it probably is safe to whitelist.  But it is annoying that you can't create any exemptions for specific hosts, that you either whitelist the entire signature or nothing at all..  

So if it was a signature for something that could sometimes be a false positive and sometimes not, the only choice you have is to keep it.. or risk an actual attack.

Meraki is good for switches, good for WiFi.  But as a Firewall, meh. 

[Dunky]

Doing reverse lookups on some of the IP's shows a mixture of many sources - windowsupdate.com, Verizon Business, Limelight Networks.  Not sure if that helps but to me it indicates these alerts are nothing to do with Kaspersky.

We don't even have Kaspersky installed anywhere.

[JessIT1]

Mainly now getting this source to one of our endpoints - a23-210-14-48.deploy.static.akamaitechnologies.com

[etb]

I'm still seeing most (all?) of the sources as what appear to be various content delivery networks (CDN's).  My #1 source today is from Edgecast.  #2 from today is HWCDN.  #8 from today is LLNW.  All of the rest of my top 10 from today reverse lookup to Level3.  (Akamai was on my top 10 for other days though.)

[Dunky]

I saw 3,500 hits yesterday with the sources being Edgecast, HWCDN and Limelight Networks.

I don't see how these can all be related to Kaspersky?????

[El-Bandito]

I can confirm one of these IP's is a windows update server.  So there's no way Meraki can tell us that this isn't a false positive.. because well there's no way microsoft's windows update server is sending us KasperSky Exploits. 

"vip0x008.map2ssl.hwcdn.net" - 209.197.3.8 =  ctldl.windowsupdate.com" 

[El-Bandito]

I looked into this exploit a bit more, and it specifically looks for .cab files.  
https://snort.org/rule_docs/1-16295

Heap-based buffer overflow in Kaspersky Antivirus (KAV) 5.0 and Kaspersky Personal Security Suite 1.1 allows remote attackers to execute arbitrary code via a CAB file with large records after the header.


Windows Update sends out .cab files through it's updates, and it can be common sometimes for antivirus / firewalls to trigger .cab files as false-positives.

[AlexP]

Hey,

This could be true if this were getting blocked by AMP, but Snort does not inspect binary payloads like .cab, .exe, etc files - it only does deep packet inspection looking for specific data patterns within the application headers.

[etb]

Very interesting - thanks, El-Bandito.

I found time to pcap all traffic for a couple minutes and search some of the IP addresses which I'm seeing logged in these IDS events.  For the couple which I have found so far, it does appear to be directly related to Microsoft Updates.  In both HTTP headers, the user-agent is shown as Microsoft-Delivery-Optimization.  In one header, the host is shown as 11.tlu.dl.delivery.mp.microsoft[.]com, and in the other header it is shown as 9.tlu.dl.delivery.mp.microsoft[.]com. 

One of the IP addresses has a reverse lookup to Edgecast, but if I search that Microsoft host, the IP address which is returned matches the one in the IDS event.  The other IP address has a reverse lookup to Highwinds Network Group, but again the Microsoft host resolves to that same IP address.

So with those limited results so far, it does seem like IDS is somehow flagging Microsoft Update traffic.  In my particular case, the mass of IDS events did happen to start on a Patch Tuesday, though that could be a coincidence.

[NickHurleyJP]

Just to add, though it's all coincidental at this point, there was recent case where Microsoft CAB files appear to be "too deep" for Cisco's deep packet inspection algorithms. Apparently, the default behavior is; if the FW cannot inspect the file, it drops the traffic. What this meant was that any attempts to update devices via windows update would end in failure. Windows update would throw generic errors and the firewall would just flag MSCAB files it couldn't upload to cloud analysis. So one is left with the choice of automatically allowing traffic through that is "x" layers too deep to be scanned or break windows updates.

[AlexP]

Hey folks,

After raising this internally with our findings, TALOS has informed us that they'll be removing this particular signature from the ruleset they share with us. To be clear, this only ever referred to a very old vulnerability from 2005 that could be triggered if a packet with a certain data structure was seen in its Layer 7 (application) headers, so we don't foresee any harm in its removal.

We cannot say with certainty given the difficulty in obtaining the packet that triggered this for everyone, but the most likely reason is just pure happenstance: a genuine service just happened to send a packet that matched on this signature without meaning to.

This could have happened because there's a genuine case for said packet whose structure was ill-handled by Kaspersky back in the day, or this simply being a signature that was too broadly defined, akin to a regular expression that's too greedy and matches more text than you want it to.

Either way, that is the full extent of what we believe can do to explain the nature of this, and prevent it from reoccurring.

[Dunky]

Great news, I'm pleased that Meraki have acted on the concerns raised by all in this thread that this was not suspicious traffic.

[Dunky]

I'm still getting thousands of these daily - is there a timeframe for whey TALOS are removing?

[AlexP]

Please let us know if you're still seeing them by EoD today - if you want to suppress the alerts, it's safe to add the signature to the bypass list in the mean time

[etb]

I'm not too concerned about the exact timing, but I'm just reporting because you asked.  I'm still seeing them as of 10:09pm EDT (2:09am UTC).  I'll check again later.

[etb]

By crazy coincidence(?), that event from Fri 6/23 @ 10:09pm EDT (2:09am UTC) was the last one of these logged.  I didn't see any of these events in question over the weekend nor today (6/26).  So it looks like this is closed out.  Thanks, AlexP and community members.

[Dunky]

Apologies for delay in responding, I have been on hols. All looks good now, thanks Alex.