IPS Snort Microsoft Windows IIS denial-of-service attempt - False positive?

[TechNick92]

Hello,

is anyone else experiencing massive problems connecting Microsoft Windows Outlook to On-Premise Exchange Servers or MS 365 since an hour (round about 11:30 am CEST) ?

All organizations with IPS activated are logging lots of events: Microsoft Windows IIS denial-of-service attempt 

Kind regards

Nick

[SimonReach]

All of our VPN users, using the client VPN, are having the issues in the UK but onsite people connecting through the MX100s are fine.

[Tortog]

Seeing the same thing, there was an option to exclude it from the IDS but it has then disappeared... 

[Rhodri]

We have the same issue. It's affecting all VPN and on-site users. I've logged a ticket with Meraki but haven't had a response yet.

[TryRebooting]

Yeah same with us. We have a couple of other systems that are also suffering, pretty much anything that is constantly cloud connected. We fix it briefly by whitelisting or changing a setting but after 10mins it breaks again and our settings on IDS disappear.

On the phone to Meraki support about it, not having a great experience with that at the moment. 

[SimonReach]

As an option, not one we'd take at the moment, but will turning off the Intrusion Detection and Prevention resolve the issue?  Anyone tried?

[Rhodri]

Haven't tried that but we're seeing thousands of dropped events on our MX100s, which could potentially indicate some kind of attack. So we don't want to disable any security related features.

[Meraki_Fan_Man]

Disabling and Enabling seems to have stopped it. Not sure if it is temporary.

Edit: lasted five minutes.

[TryRebooting]

Dropping it to detection from prevention does seem to allow a reconnect.

[CJHarms]

Same for us. Thousands of Events regarding the Snort Rule mentioned in the first Post.

I have switched to Detection instead of Prevention for now till this is fixed. So far "disabling" the IDS or putting it into Detection Mode only seems to fix the Problem.

[TryRebooting]

Switching back to Prevention just breaks things again after 5mins so yeah currently detection mode seems to be the soft fix for now.

[EricI]

Yes, same problem here causing massive problems for Microsoft desktop applications unable to login using TLS 1.2. We whitelisted the Snort rule (Sid 1-60381) and reported a false positive to snort.org 

[Rhodri]

How do you know it's a false-positive?

[EricI]

-Our own clients access to our own services were affected - only desktop applications, browser access was fine.
-No Endpoint security related issues are reported from our endpoint security stack.
-No issues are reported from the Microsoft security services used.
-Clients that pass through other IPS / Threat detection services than the Meraki / Snort combo report no similar issues.

[TechNick92]

I think there is some correlation to latest Exchange-server updates, where Microsoft recommended to activate Extended Protection. This changes a lot in the SSL configuation of the IIS. And maybe someone on snort tried to create a IDS rule to block this kind of attack and finally configured some bad preferences to identify a real attack instead of normal traffic to an IIS with SSL

[SimonReach]

We're also having issues with Windows Update and our Dell Command Update application as well.

[RobertMiranda]

Same issue with Windows Update. Outlook and OneDrive not working as well.

[GarethR]

This is due to a SNORT definition update seen here: https://snort.org/rule_docs/1-60381
This is a result of a CVE reported by Microsoft seen here: https://msrc.microsoft.com/update-guide/en-us/vulnerability/CVE-2022-35748

The recommendation by Microsoft is to patch the required software. In the meantime you can whitelist the SNORT rule 1-60381 from the Threat Protection centre however this must be understood that you may be vulnerable to the CVE reported.

[halotpg]

The issue could be related to the ongoing Microsoft incident EX411786. Our current working theory is that Microsoft apps traffic (desktop clients only) is being interpreted as DoS attempts and blocked.

[Dunky]

Microsoft have acknowledged a service heath issue - EX411786

[JessIT1]

Same issues here only our MX84 is affected by these IDS Microsoft Windows IIS denial-of-service attempt

Blocked Outlook on desktops, blocked RDGateway server access, VPN.

Setting the MX84 to detection from prevention is a temporary fix. Actions are now being logged as allowed, most destinations are Google Maps, Amazon CDN's, etc.

[martin-netx]

 We are also seeing lots of "Microsoft Windows IIS denial-of-service attempt" occurances on multiple MXs. Appears to impact on google drive too if using the app as opposed to browser. 

[martin-netx]

Just off the phone with Meraki TAC, engineer was very helpful. They have just now added an IPS rule called OS-Microsoft Windows IIS denial-of-service attempt 1:60381. We should be able to allow this rule if we wish as a form of workaround for the moment. 

Seems that it is a Microsoft update related issue but I'm not too clear on it yet.

[JessIT1]

I had our MX84 set to detection temporarily from prevention, are you saying I could put back to prevention then turn on whitelist for Rule ID1-60381 ?  thanks

[martin-netx]

Hi JessIT1, I'm just waiting to hear if the IDS allow has worked for actual users...

[Wifikohai]

Could you explain how to do that?

Thanks

Regards

[Rhodri]

If you have an MX100 then go to "Security & SD WAN" then "threat protection". Under "Intrusion detection and prevention", add a rule and search for "1:60381". Save changes.

You should read the Microsoft CVE and snort rule before doing this, so you can determine whether your environment is actually vulnerable to this exploit.

https://msrc.microsoft.com/update-guide/en-us/vulnerability/CVE-2022-35748

https://snort.org/rule_docs/1-60381

[TechNick92]

Reply of Meraki support: IPS/SNORT blocking Microsoft traffic

Dear Giacomo,

i can't accept this explanation of Meraki.

There are even problems with big services like Microsoft Exchange Online and so on. So nobody is able to fix this beside Microsoft themselves. 

I understand there is some kind of security issue but it was released hours ago and now breaking traffic for a lot of customers is not the right way to handle it.

Does the whitlisting work or is it dropped as there answers already said after 5-10 minutes? Turning off the IDS is NO solution

[Rhodri]

We've added an allow rule on our MX100s for this particular type of traffic. It's on the list of rules as "1:60381". This has resolved the problem. We need more information from Microsoft on the potential ongoing threat/mitigation.

[Wifikohai]

Hi Rhodri,

Please I don´t were to put this rule. Dis you put by youself or you had to call meraki? Could you tell me where is this option? 

Thanks in advance.

Regards

[Rhodri]

Hi. Here you go...

https://community.meraki.com/t5/Security-SD-WAN/IPS-Snort-Microsoft-Windows-IIS-denial-of-service-at...

[41D5]

What we found:

Meraki SDWAN appliance with IPS prevention enabled.

In Security Center, we see this alert:

Our resolution:

Whitelist SNORT Signature 1:60381 (Click "On" to whitelist)

At this point, all of your Office 365 / Internet / Outlook / MS Teams issues should be resolved.  Users should be working.  The users may need to restart apps or reboot.

Then patch all Microsoft OS's.  You can't patch until the rule is Whitelisted.

After everything is patched, enable the SNORT signature 1:60381 (Click "Off" to remove from whitelist):

This has worked for 3 organizations where we implemented this fix.

---UPDATE---

I need to eat a nice plate of crow.  While the fix to whitelist the snort rule works 100%, applying the Windows Updates did not resolve the issue.  When we turn on the SNORT signature, it breaks most clients again.  We thought the Windows Updates fixed it, but it turned out that after some reboots and resets, the applications are still being blocked with the Whitelist disabled.  We also confirmed this in Security Center as we still see incrementing hits on the SNORT rule.

So we are leaving the Whitelist ON for now.

[Rhodri]

Is there a patch for Windows 10? The Microsoft CVE only refers to Windows Server operating systems.

[AxPayne]

I haven't found one as of yet. I've been looking. I checked my machine and only had this cumulative update available. KB5016616

[41D5]

To be totally honest, we just ran Windows Update and ensured all of the August 9th patches were applied.  While we thought this fixed the issue with the SNORT signature, it did not.  The client systems may appear to work for some time after the SNORT signature is enabled (Whitelist set to OFF), but the client systems will break after a reboot or after some time.

Keep Whitelist ON for now for SNORT Signature 1:60381

[BrianMorris]

I applied this Whitelist ON change in all of our Meraki routers and it seems to have fixed our ability to remotely access their computers and Outlook and OneDrive.

[Brandon123s]

The KB patch will not fix this, as the patch only prevents the exploit from working.

Blocking of legitimate traffic is simply a false positive. Until they update the rule it will always block legitimate TLS 1.2 handshakes that happen too frequently in accordance to how the rule is implemented.

Patch + whitelist is the only method that will work and keep us protected until meraki fixes the rule.

[Kevin-C]

Overnight we had 35000 instances of the Microsoft Windows IIS denial-of-service attempt.  The weird part was that we were having lots of internal communication problems.  I could ping devices at a branch, but they couldn't get to some internet sites or internal web sites.  We route all our traffic back though our data centers to be filtered by our firewalls.  The problems only went away when I moved the IDS to detection mode instead of prevention.

[Brandon123s]

This is because the exploit mitigation rule targets too frequent TLS 1.2 hellos. Anything using TLS 1.2 could be blocked and some non-microsoft services were at my org according to the logs.

[Jamest201]

Anyone know the KB of the patch? I'm fully patched now and put IDS back to prevention with the rule allowed and everything broke again

[DBeiner]

August 9, 2022—KB5016623 (OS Build 17763.3287) (microsoft.com)

[Brandon123s]

Upon applying the rule it took a few minutes for it to take effect for my org. Within 5 minutes the whitelist was recognized and there were no more issues.

You get this working?

[Rhodri]

The patch only applies to servers. It won't prevent Meraki traffic being blocked from "endpoints that are leveraging TLS 1.2", as Microsoft put it.

I think we need to wait for Microsoft to close out the incident - MO411804. They're "engaging with their firewall partners" so presumably the snork rule will be corrected at some point.

[nlev]

Latest update on the MO411804 incident:

August 10, 2022 1:56 PM · Quick update
The firewall partner is currently reviewing options to remediate impact.

This quick update is designed to give the latest information on this issue.

[TMTECH]

this is the latest alert from MS

August 10, 2022 10:28 AM · Quick update

We're looking at recent changes made within the Microsoft-managed infrastructure and reviewing endpoints that are leveraging TLS 1.2. Additionally, we're contacting firewall partners to assist our investigation. This quick update is designed to give the latest information on this issue.

August 10, 2022 9:53 AM

Title: Some users may be unable to connect to multiple Microsoft 365 services. User Impact: Users may be unable to connect to multiple Microsoft 365 services. More info: Impacted services include, but are not limited to: - Outlook desktop client - OneDrive for Business - Microsoft Teams Current status: After analyzing system telemetry and Fiddler logs from impacted users, we suspect that third-party firewall devices are potentially blocking legitimate Microsoft traffic. Affected customers have reported that disabling firewall rules blocking TLS 1.2 is mitigating impact. We're continuing our investigation into the underlying cause. Scope of impact: At this time, impact appears to be specific to some users who are served through the affected infrastructure. Next update by: Wednesday, August 10, 2022, 11:30 AM (3:30 PM UTC)

[haupt]

Meraki Support is aware of the issue. Please view this post Microsoft vulnerability and IPS/SNORT and subscribe to the post for updates. 

[MOD NOTE: Marking this reply as the solution for greater visibility. Please refer to the Services Notices post linked above (or here: link) for the latest updates from Meraki on this issue]

[Brandon123s]

I don't agree with the resolution of this issue from Meraki. 

Can you confirm that a 100% patched environment does not suffer from the false positive detections? A few people in this thread seem to have stated the false positive detections are still happening despite patching.

As I understand it the only way to be protected from this not yet seen in the wild exploit and still have TLS 1.2 working is to whitelist this rule and patch systems.

edit: I believe this is now resolved as the affected SNORT rule has been adjusted. At least one post here says this has resolved the issue after re-enabling the rule.

[nlev]

Agreed, I don't think this should be marked as solved just yet.

Edit: Sorry, I see it was only marked as solved for visibility, not actually considered solved.

[Rhodri]

You are correct. Patching does not prevent false positives and the blocking of traffic.

[41D5]

@Brandon123s  I totally agree.  The Microsoft vulnerability and IPS/SNORT post makes it sound like the issue is resolved or to call Meraki support.  It should be make clear at this time, that this is an active issue with no resolution.  The workaround is to whitelist the signature.

The advisory from Microsoft posted by @TMTECH is much closer to reality.

[Rhodri]

This isn't a solution. Patching endpoints does not prevent the offending traffic and it doesn't prevent it from being blocked.

You cannot currently fix the problem without either disabling IDS or by creating a rule.

[SteveBradbury]

Definitely not solved, fix/remove the snort rule please.

[JessIT1]

I added whitelist for Rule ID1-60381, can anyone confirm if then putting threat protection back to prevention from detection (a temp fix this morning) will not break connections again..?  thanks

[BrianMorris]

I left my sites on Prevention and just turned the Whitelist setting to ON for this and all of our problems disappeared (OneDrive app, Outlook app, Splashtop, Syncro remote access).

[martin-netx]

We've now got multiple MXs with IDS set to prevention and the whitelisting of Rule ID1-60381. All looks to be working OK now.

[JessIT1]

After adding whitelist for Rule ID 1-60381 it's showing null not sure if that means it's working.. however I turned IDS back to prevention just as a precaution, did not want to leave on detection, so far desktop Outlook staying connected, VPN connections not dropped.

[nlev]

Same here. It displays null, but everything is working.

[Rhodri]

Yeah, it says "null" after you apply it but it does work.

[Eclipse]

I don't trust the whitelist because of the odd traffic coming through. We are getting hammered by this address below and I don't want to allow this. Does anyone else see this as well?

[Brandon123s]

the CVE itself says there is no known exploits for this in the wild. The CVE is from yesterday and it's all about TLS 1.2 hellos.

Anything using TLS 1.2 could be affected here. Tons of companies use amazon AWS, so the above whois probably some vendor hosted service that people authenticate against.

I also have clients in my network connecting to similar aws compute resources and reporting the same thing. The traffic being blocked though is initiated by clients within my LAN, not from the outside. In your security center events log, are you seeing incoming traffic being blocked from that address?

[BrianMorris]

Here's the first user that called me this morning -- these are addresses that were blocked with this SNORT rule. 

[Eclipse]

This is what I'm seeing today.

[Sugarmill_Russ]

This is what I'm seeing.  What concerns me is that the majority are "unknown" affected OS's...

[Eclipse]

Meraki isn't great at determining the OS of each client unless you are 100% Meraki gear. 

[Eclipse]

Yes. That address is where the majority of the DoD events are coming from which is triggering the issue on our end. I've never seen this much traffic from AWS before and was wondering if anyone has seen something similar. I have emailed their abuse center for what it's worth.

[Meraki_Fan_Man]

So no patch for Windows 10?
Rule with Prevention is still blocking for us.

[WadeAlsup]

Same. Rule is set to "whitelist" but shows as null. Others say it is working, but still showing blocked events for us.

[Rhodri]

What device do you have?

[WadeAlsup]

MX 100 at our primary site.

[Rhodri]

Not sure if it helps but ours is set like this and it works.

[Meraki_Fan_Man]

Yes. That still shows blocking in Events.
MX250 Current FW 17.7