IPS Snort Microsoft Windows IIS denial-of-service attempt - False positive?

[TechNick92]

Hello,

is anyone else experiencing massive problems connecting Microsoft Windows Outlook to On-Premise Exchange Servers or MS 365 since an hour (round about 11:30 am CEST) ?

All organizations with IPS activated are logging lots of events: Microsoft Windows IIS denial-of-service attempt 

Kind regards

Nick

[SimonReach]

All of our VPN users, using the client VPN, are having the issues in the UK but onsite people connecting through the MX100s are fine.

[Tortog]

Seeing the same thing, there was an option to exclude it from the IDS but it has then disappeared... 

[Rhodri]

We have the same issue. It's affecting all VPN and on-site users. I've logged a ticket with Meraki but haven't had a response yet.

[TryRebooting]

Yeah same with us. We have a couple of other systems that are also suffering, pretty much anything that is constantly cloud connected. We fix it briefly by whitelisting or changing a setting but after 10mins it breaks again and our settings on IDS disappear.

On the phone to Meraki support about it, not having a great experience with that at the moment. 

[SimonReach]

As an option, not one we'd take at the moment, but will turning off the Intrusion Detection and Prevention resolve the issue?  Anyone tried?

[Rhodri]

Haven't tried that but we're seeing thousands of dropped events on our MX100s, which could potentially indicate some kind of attack. So we don't want to disable any security related features.

[Meraki_Fan_Man]

Disabling and Enabling seems to have stopped it. Not sure if it is temporary.

Edit: lasted five minutes.

[TryRebooting]

Dropping it to detection from prevention does seem to allow a reconnect.

[CJHarms]

Same for us. Thousands of Events regarding the Snort Rule mentioned in the first Post.

I have switched to Detection instead of Prevention for now till this is fixed. So far "disabling" the IDS or putting it into Detection Mode only seems to fix the Problem.

[TryRebooting]

Switching back to Prevention just breaks things again after 5mins so yeah currently detection mode seems to be the soft fix for now.

[EricI]

Yes, same problem here causing massive problems for Microsoft desktop applications unable to login using TLS 1.2. We whitelisted the Snort rule (Sid 1-60381) and reported a false positive to snort.org 

[Rhodri]

How do you know it's a false-positive?

[EricI]

-Our own clients access to our own services were affected - only desktop applications, browser access was fine.
-No Endpoint security related issues are reported from our endpoint security stack.
-No issues are reported from the Microsoft security services used.
-Clients that pass through other IPS / Threat detection services than the Meraki / Snort combo report no similar issues.

[TechNick92]

I think there is some correlation to latest Exchange-server updates, where Microsoft recommended to activate Extended Protection. This changes a lot in the SSL configuation of the IIS. And maybe someone on snort tried to create a IDS rule to block this kind of attack and finally configured some bad preferences to identify a real attack instead of normal traffic to an IIS with SSL

[SimonReach]

We're also having issues with Windows Update and our Dell Command Update application as well.

[RobertMiranda]

Same issue with Windows Update. Outlook and OneDrive not working as well.

[GarethR]

This is due to a SNORT definition update seen here: https://snort.org/rule_docs/1-60381
This is a result of a CVE reported by Microsoft seen here: https://msrc.microsoft.com/update-guide/en-us/vulnerability/CVE-2022-35748

The recommendation by Microsoft is to patch the required software. In the meantime you can whitelist the SNORT rule 1-60381 from the Threat Protection centre however this must be understood that you may be vulnerable to the CVE reported.

[halotpg]

The issue could be related to the ongoing Microsoft incident EX411786. Our current working theory is that Microsoft apps traffic (desktop clients only) is being interpreted as DoS attempts and blocked.

[Dunky]

Microsoft have acknowledged a service heath issue - EX411786

[JessIT1]

Same issues here only our MX84 is affected by these IDS Microsoft Windows IIS denial-of-service attempt

Blocked Outlook on desktops, blocked RDGateway server access, VPN.

Setting the MX84 to detection from prevention is a temporary fix. Actions are now being logged as allowed, most destinations are Google Maps, Amazon CDN's, etc.

[martin-netx]

 We are also seeing lots of "Microsoft Windows IIS denial-of-service attempt" occurances on multiple MXs. Appears to impact on google drive too if using the app as opposed to browser. 

[martin-netx]

Just off the phone with Meraki TAC, engineer was very helpful. They have just now added an IPS rule called OS-Microsoft Windows IIS denial-of-service attempt 1:60381. We should be able to allow this rule if we wish as a form of workaround for the moment. 

Seems that it is a Microsoft update related issue but I'm not too clear on it yet.

[JessIT1]

I had our MX84 set to detection temporarily from prevention, are you saying I could put back to prevention then turn on whitelist for Rule ID1-60381 ?  thanks

[martin-netx]

Hi JessIT1, I'm just waiting to hear if the IDS allow has worked for actual users...

[Wifikohai]

Could you explain how to do that?

Thanks

Regards

[Rhodri]

If you have an MX100 then go to "Security & SD WAN" then "threat protection". Under "Intrusion detection and prevention", add a rule and search for "1:60381". Save changes.

You should read the Microsoft CVE and snort rule before doing this, so you can determine whether your environment is actually vulnerable to this exploit.

https://msrc.microsoft.com/update-guide/en-us/vulnerability/CVE-2022-35748

https://snort.org/rule_docs/1-60381

[TechNick92]

Reply of Meraki support: IPS/SNORT blocking Microsoft traffic

Dear Giacomo,

i can't accept this explanation of Meraki.

There are even problems with big services like Microsoft Exchange Online and so on. So nobody is able to fix this beside Microsoft themselves. 

I understand there is some kind of security issue but it was released hours ago and now breaking traffic for a lot of customers is not the right way to handle it.

Does the whitlisting work or is it dropped as there answers already said after 5-10 minutes? Turning off the IDS is NO solution

[Rhodri]

We've added an allow rule on our MX100s for this particular type of traffic. It's on the list of rules as "1:60381". This has resolved the problem. We need more information from Microsoft on the potential ongoing threat/mitigation.

[Wifikohai]

Hi Rhodri,

Please I don´t were to put this rule. Dis you put by youself or you had to call meraki? Could you tell me where is this option? 

Thanks in advance.

Regards

[Rhodri]

Hi. Here you go...

https://community.meraki.com/t5/Security-SD-WAN/IPS-Snort-Microsoft-Windows-IIS-denial-of-service-at...

[41D5]

What we found:

Meraki SDWAN appliance with IPS prevention enabled.

In Security Center, we see this alert:

Our resolution:

Whitelist SNORT Signature 1:60381 (Click "On" to whitelist)

At this point, all of your Office 365 / Internet / Outlook / MS Teams issues should be resolved.  Users should be working.  The users may need to restart apps or reboot.

Then patch all Microsoft OS's.  You can't patch until the rule is Whitelisted.

After everything is patched, enable the SNORT signature 1:60381 (Click "Off" to remove from whitelist):

This has worked for 3 organizations where we implemented this fix.

---UPDATE---

I need to eat a nice plate of crow.  While the fix to whitelist the snort rule works 100%, applying the Windows Updates did not resolve the issue.  When we turn on the SNORT signature, it breaks most clients again.  We thought the Windows Updates fixed it, but it turned out that after some reboots and resets, the applications are still being blocked with the Whitelist disabled.  We also confirmed this in Security Center as we still see incrementing hits on the SNORT rule.

So we are leaving the Whitelist ON for now.

[Rhodri]

Is there a patch for Windows 10? The Microsoft CVE only refers to Windows Server operating systems.

[AxPayne]

I haven't found one as of yet. I've been looking. I checked my machine and only had this cumulative update available. KB5016616

[41D5]

To be totally honest, we just ran Windows Update and ensured all of the August 9th patches were applied.  While we thought this fixed the issue with the SNORT signature, it did not.  The client systems may appear to work for some time after the SNORT signature is enabled (Whitelist set to OFF), but the client systems will break after a reboot or after some time.

Keep Whitelist ON for now for SNORT Signature 1:60381

[BrianMorris]

I applied this Whitelist ON change in all of our Meraki routers and it seems to have fixed our ability to remotely access their computers and Outlook and OneDrive.

[Brandon123s]

The KB patch will not fix this, as the patch only prevents the exploit from working.

Blocking of legitimate traffic is simply a false positive. Until they update the rule it will always block legitimate TLS 1.2 handshakes that happen too frequently in accordance to how the rule is implemented.

Patch + whitelist is the only method that will work and keep us protected until meraki fixes the rule.

[Kevin-C]

Overnight we had 35000 instances of the Microsoft Windows IIS denial-of-service attempt.  The weird part was that we were having lots of internal communication problems.  I could ping devices at a branch, but they couldn't get to some internet sites or internal web sites.  We route all our traffic back though our data centers to be filtered by our firewalls.  The problems only went away when I moved the IDS to detection mode instead of prevention.

[Brandon123s]

This is because the exploit mitigation rule targets too frequent TLS 1.2 hellos. Anything using TLS 1.2 could be blocked and some non-microsoft services were at my org according to the logs.

[Jamest201]

Anyone know the KB of the patch? I'm fully patched now and put IDS back to prevention with the rule allowed and everything broke again

[DBeiner]

August 9, 2022—KB5016623 (OS Build 17763.3287) (microsoft.com)

[Brandon123s]

Upon applying the rule it took a few minutes for it to take effect for my org. Within 5 minutes the whitelist was recognized and there were no more issues.

You get this working?

[Rhodri]

The patch only applies to servers. It won't prevent Meraki traffic being blocked from "endpoints that are leveraging TLS 1.2", as Microsoft put it.

I think we need to wait for Microsoft to close out the incident - MO411804. They're "engaging with their firewall partners" so presumably the snork rule will be corrected at some point.

[nlev]

Latest update on the MO411804 incident:

August 10, 2022 1:56 PM · Quick update
The firewall partner is currently reviewing options to remediate impact.

This quick update is designed to give the latest information on this issue.

[TMTECH]

this is the latest alert from MS

August 10, 2022 10:28 AM · Quick update

We're looking at recent changes made within the Microsoft-managed infrastructure and reviewing endpoints that are leveraging TLS 1.2. Additionally, we're contacting firewall partners to assist our investigation. This quick update is designed to give the latest information on this issue.

August 10, 2022 9:53 AM

Title: Some users may be unable to connect to multiple Microsoft 365 services. User Impact: Users may be unable to connect to multiple Microsoft 365 services. More info: Impacted services include, but are not limited to: - Outlook desktop client - OneDrive for Business - Microsoft Teams Current status: After analyzing system telemetry and Fiddler logs from impacted users, we suspect that third-party firewall devices are potentially blocking legitimate Microsoft traffic. Affected customers have reported that disabling firewall rules blocking TLS 1.2 is mitigating impact. We're continuing our investigation into the underlying cause. Scope of impact: At this time, impact appears to be specific to some users who are served through the affected infrastructure. Next update by: Wednesday, August 10, 2022, 11:30 AM (3:30 PM UTC)

[haupt]

Meraki Support is aware of the issue. Please view this post Microsoft vulnerability and IPS/SNORT and subscribe to the post for updates. 

[MOD NOTE: Marking this reply as the solution for greater visibility. Please refer to the Services Notices post linked above (or here: link) for the latest updates from Meraki on this issue]

[Brandon123s]

I don't agree with the resolution of this issue from Meraki. 

Can you confirm that a 100% patched environment does not suffer from the false positive detections? A few people in this thread seem to have stated the false positive detections are still happening despite patching.

As I understand it the only way to be protected from this not yet seen in the wild exploit and still have TLS 1.2 working is to whitelist this rule and patch systems.

edit: I believe this is now resolved as the affected SNORT rule has been adjusted. At least one post here says this has resolved the issue after re-enabling the rule.

[nlev]

Agreed, I don't think this should be marked as solved just yet.

Edit: Sorry, I see it was only marked as solved for visibility, not actually considered solved.

[Rhodri]

You are correct. Patching does not prevent false positives and the blocking of traffic.

[41D5]

@Brandon123s  I totally agree.  The Microsoft vulnerability and IPS/SNORT post makes it sound like the issue is resolved or to call Meraki support.  It should be make clear at this time, that this is an active issue with no resolution.  The workaround is to whitelist the signature.

The advisory from Microsoft posted by @TMTECH is much closer to reality.

[Rhodri]

This isn't a solution. Patching endpoints does not prevent the offending traffic and it doesn't prevent it from being blocked.

You cannot currently fix the problem without either disabling IDS or by creating a rule.

[SteveBradbury]

Definitely not solved, fix/remove the snort rule please.

[JessIT1]

I added whitelist for Rule ID1-60381, can anyone confirm if then putting threat protection back to prevention from detection (a temp fix this morning) will not break connections again..?  thanks

[BrianMorris]

I left my sites on Prevention and just turned the Whitelist setting to ON for this and all of our problems disappeared (OneDrive app, Outlook app, Splashtop, Syncro remote access).

[martin-netx]

We've now got multiple MXs with IDS set to prevention and the whitelisting of Rule ID1-60381. All looks to be working OK now.

[JessIT1]

After adding whitelist for Rule ID 1-60381 it's showing null not sure if that means it's working.. however I turned IDS back to prevention just as a precaution, did not want to leave on detection, so far desktop Outlook staying connected, VPN connections not dropped.

[nlev]

Same here. It displays null, but everything is working.

[Rhodri]

Yeah, it says "null" after you apply it but it does work.

[Eclipse]

I don't trust the whitelist because of the odd traffic coming through. We are getting hammered by this address below and I don't want to allow this. Does anyone else see this as well?

[Brandon123s]

the CVE itself says there is no known exploits for this in the wild. The CVE is from yesterday and it's all about TLS 1.2 hellos.

Anything using TLS 1.2 could be affected here. Tons of companies use amazon AWS, so the above whois probably some vendor hosted service that people authenticate against.

I also have clients in my network connecting to similar aws compute resources and reporting the same thing. The traffic being blocked though is initiated by clients within my LAN, not from the outside. In your security center events log, are you seeing incoming traffic being blocked from that address?

[BrianMorris]

Here's the first user that called me this morning -- these are addresses that were blocked with this SNORT rule. 

[Eclipse]

This is what I'm seeing today.

[Sugarmill_Russ]

This is what I'm seeing.  What concerns me is that the majority are "unknown" affected OS's...

[Eclipse]

Meraki isn't great at determining the OS of each client unless you are 100% Meraki gear. 

[Eclipse]

Yes. That address is where the majority of the DoD events are coming from which is triggering the issue on our end. I've never seen this much traffic from AWS before and was wondering if anyone has seen something similar. I have emailed their abuse center for what it's worth.

[Meraki_Fan_Man]

So no patch for Windows 10?
Rule with Prevention is still blocking for us.

[WadeAlsup]

Same. Rule is set to "whitelist" but shows as null. Others say it is working, but still showing blocked events for us.

[Rhodri]

What device do you have?

[WadeAlsup]

MX 100 at our primary site.

[Rhodri]

Not sure if it helps but ours is set like this and it works.

[Meraki_Fan_Man]

Yes. That still shows blocking in Events.
MX250 Current FW 17.7

[DaveSP1]

Experiencing the same on our MX84. Had to switch from prevent to detect and working this way right now.

[Larrie]

We have set IDS to detect and got the null: 160381 whitelist rule in place but Microsoft VPN still appears to be blocked - does our MX84 need restarting or power cycling?

[halpmeh]

Is anyone else getting a ton of "Microsoft wimgapi LoadIntegrityInfo heap buffer overflow attempt" blocks after allowing "Microsoft Windows IIS denial-of-service attempt" on their MX?

Seems off.

[BrianMorris]

The latest from Microsoft: 

August 10, 2022 1:02 PM

Title: Some users may be unable to connect to multiple Microsoft 365 services. User Impact: Users may be unable to connect to multiple Microsoft 365 services. More info: Impacted services include, but are not limited to: - Outlook desktop client - OneDrive for Business - Microsoft Teams Affected customers have reported that disabling firewall rules blocking TLS 1.2 is mitigating impact. Current status: We continue to work with the firewall partner to investigate a Snort rule which is contributing to impact. Our focus remains on mitigation and from user reports, disabling the specific firewall rule provides immediate relief. Additionally, we continue to investigate recent changes within the Microsoft-managed environment to rule out potential causes of impact. Scope of impact: At this time, impact appears to be specific to some users who are served through the affected infrastructure. Next update by: Wednesday, August 10, 2022, 2:30 PM (6:30 PM UTC)

[Larrie]

We have set IDS to detect from prevent and got the null: 160381 whitelist rule in place but Microsoft VPN still appears to be blocked - does our MX84 need restarting or power cycling?

[BrianMorris]

Here's what mine looks like. Make sure that you apply it to all sites if you are adding manually. I took the easy route and went to Security Center, it shows all sites and I just clicked on "Microsoft Windows IIS denial-of-service attempt" and clicked Whitelist ON.

No reboots needed!

[Larrie]

Yes that's what we have set on MX84 - all devices onsite now able to connect outbound to Microsoft services e.g. outlook to Exchange online, etc

But when trying to connect to Microsoft RDS VPN system not connecting from offsite

Puzzled

[nlev]

Perhaps your Microsoft RDS VPN issue is unrelated to this IDS rule. Do you see any other rules in the security center relating to the impacted clients?

[MarcAEC]

It's really disappointing that Meraki didn't push out a fix when the details of this first emerged.  Instead, each customer whose email was nonfunctional was left to figure out the problem and manually add a threat protection override.  What's the point of something being in the cloud if the vendor can't act fast when there is an obvious and serious problem?

The irony is that the vulnerability allows someone to conduct a denial of service attack, and the result of Meraki protecting from it is a denial of service.

[DetroitJeremy]

I only added the allow list entry to rule to a couple networks but now see it on more.  Did Meraki add that entry themselves as a work around while things get worked out between MS, SNORT and Meraki?

[nlev]

No, we saw the same thing and others have been reporting that as well. Adding it to one network seems to unexpectedly add it to all networks in the organization.

[Larrie]

yes we were on to meraki support this morning - they didn't mention this issue,  they told us we had hardware issue and to swap in our backup MX84 - very poor

[Wolfpackpetty]

3 out 6 of our devices were stopping traffic due to SNORT 1-60381. Cannot get any answers as to why whitelist a working rule. Now the rule has been deleted from SNORT all together? What was found to be wrong with the rule? 

[nlev]

The official update says: "The Snort rules have been removed to reduce the impact. "  Was there another rule other than 1-60381 that was contributing to the issues?

[SomeITGuy]

I am still seeing traffic blocked by 1-60381 SNORT on my MX100

[Network-Dude]

SomeITGuy... Did the issue clear up the matter after the change?  We have some MX100s within the environment and the solution seemed to work for us.

[Larrie]

difficult to tell if it was us changing and rebooting MX and RDS gateway server or if MX84 received Meraki update removing that IDS snort rule - but we do seem stable now and MS VPN is working again.... a whole day of pain that I think Meraki could have been quicker and better at communicating to its customers

[Larrie]

What Meraki have not communicated is if we can change IDS back from detect to prevent - has Meraki made a statement?

[DetroitJeremy]

The offending SNORT rule has been removed and Meraki stated things should be fully resolved by 3PM PDT.  So I'd say the conservative approach would be to wait 30 minutes then switch IDS to prevent.  I allow-listed the offending rule and stayed on prevent - for me this resolved the issue almost immediately.

[Network-Dude]

I wish Meraki would have posted the information on the dashboard early this morning.

[Meraki_Fan_Man]

Is it true if your Windows servers and Windows 10 PCs were on the latest OS level AND had the patch from 8/9 the traffic would NOT have been blocked?

[Eclipse]

No. Meraki was blocking traffic regardless.

[JessIT1]

So at this point with my MX84 threat protection set back to prevention and Meraki removing the Snort rule, should we remove the whitelist rule for 1:60381 in our Intrusion detection and prevention?

[Grumples]

Anybody else having new problems today?  We added the Allow rule for 1:60381 to our Threat Protection page to clear up the issues, and that rule is still there today.  But...about 2 hours ago, we started having all sorts or problems again.  We were unable to visit some webpages.  We had back-end processes on servers that communicate with Web APIs that no longer work.

Thinking of how we had bizarre problems yesterday, I set our IPS mode to DETECTION and all of our problems went away a few minutes later.  So clearly the IPS in the MX100 was dropping traffic.  But I haven't been able to determine anything about the IPS rule.  When I search our Events list for all IPS items, I get an empty list returns. It seems crazy to me that we have no events for IPS, not even a Rule Update event.

Anyone else out there having new problems with the IPS today?

[EricI]

Nope. No reports, no indications of anything similar to yesterday where all sites were affected and IPS events during the time frame increased from a normal of a few hundred to tens of thousands.

[Dunky]

No issues here, Still have the rule whitelisted (will remove next week). No unusual IDS events showing in Security Center.

[Larrie]

Same, no further problems, but like you left the whitelist one in, to change at later date

[Grumples]

In Security Center, we've gotten a few of these events:

SERVER-OTHER Exim unauthenticated remote code execution attempt

Rule ID1-53376

[AmyReyes · Accepted solution]

Hi folks! Please be aware that this issue has been resolved as of today (5am GMT August 11, 2022). More info can be found on the Service Notices page here.

If you're still experiencing trouble, please treat it as a different issue and contact Meraki Support. Thank you!

[billyzoellers]

How do we go about getting a root cause analysis for this? I appreciate that it has been resolved, but there are certainly some unanswered questions. A few I can think of:

• Why did this rule seem to impact more traffic than it was intended to?
• How can we review the specific behavior(s) that the rule was matching on?
• What's the review process for new IPS/IDS rules being added?
• How can customers be aware when a new rule is being added?

[AmyReyes]

Hi @billyzoellers, those are great questions. Currently, this is under investigation still. Once we have more information to share, we will update everyone. Stay tuned!

[martin-netx]

I see that if we follow the link to the rule details in Snort this rule now shows as "deleted". I guess this means we can switch off the whitelisting of this rule if it has actually been deleted?

[Dunky]

I am removing our whitelisted rule tomorrow, I don't expect any issues.