IPS Snort Microsoft Windows IIS denial-of-service attempt - False positive?

Solved
TechNick92
Here to help

IPS Snort Microsoft Windows IIS denial-of-service attempt - False positive?

Hello,

 

is anyone else experiencing massive problems connecting Microsoft Windows Outlook to On-Premise Exchange Servers or MS 365 since an hour (round about 11:30 am CEST) ?

 

All organizations with IPS activated are logging lots of events: Microsoft Windows IIS denial-of-service attempt 

 

Kind regards

Nick

1 Accepted Solution
AmyReyes
Community Manager
Community Manager

Hi folks! Please be aware that this issue has been resolved as of today (5am GMT August 11, 2022). More info can be found on the Service Notices page here.

If you're still experiencing trouble, please treat it as a different issue and contact Meraki Support. Thank you!

View solution in original post

103 Replies 103
SimonReach
Getting noticed

All of our VPN users, using the client VPN, are having the issues in the UK but onsite people connecting through the MX100s are fine.

Tortog
New here

Seeing the same thing, there was an option to exclude it from the IDS but it has then disappeared... 

Rhodri
New here

We have the same issue. It's affecting all VPN and on-site users. I've logged a ticket with Meraki but haven't had a response yet.

TryRebooting
Just browsing

Yeah same with us. We have a couple of other systems that are also suffering, pretty much anything that is constantly cloud connected. We fix it briefly by whitelisting or changing a setting but after 10mins it breaks again and our settings on IDS disappear.

 

On the phone to Meraki support about it, not having a great experience with that at the moment. 

SimonReach
Getting noticed

As an option, not one we'd take at the moment, but will turning off the Intrusion Detection and Prevention resolve the issue?  Anyone tried?

Rhodri
New here

Haven't tried that but we're seeing thousands of dropped events on our MX100s, which could potentially indicate some kind of attack. So we don't want to disable any security related features.

Meraki_Fan_Man
Here to help

Disabling and Enabling seems to have stopped it. Not sure if it is temporary.

Edit: lasted five minutes.

TryRebooting
Just browsing

Dropping it to detection from prevention does seem to allow a reconnect.

 

CJHarms
Here to help

Same for us. Thousands of Events regarding the Snort Rule mentioned in the first Post.

 

I have switched to Detection instead of Prevention for now till this is fixed. So far "disabling" the IDS or putting it into Detection Mode only seems to fix the Problem.

TryRebooting
Just browsing

Switching back to Prevention just breaks things again after 5mins so yeah currently detection mode seems to be the soft fix for now.

EricI
Here to help

Yes, same problem here causing massive problems for Microsoft desktop applications unable to login using TLS 1.2. We whitelisted the Snort rule (Sid 1-60381) and reported a false positive to snort.org 

Rhodri
New here

How do you know it's a false-positive?

EricI
Here to help

-Our own clients access to our own services were affected - only desktop applications, browser access was fine.
-No Endpoint security related issues are reported from our endpoint security stack.
-No issues are reported from the Microsoft security services used.
-Clients that pass through other IPS / Threat detection services than the Meraki / Snort combo report no similar issues.

TechNick92
Here to help

I think there is some correlation to latest Exchange-server updates, where Microsoft recommended to activate Extended Protection. This changes a lot in the SSL configuation of the IIS. And maybe someone on snort tried to create a IDS rule to block this kind of attack and finally configured some bad preferences to identify a real attack instead of normal traffic to an IIS with SSL

SimonReach
Getting noticed

We're also having issues with Windows Update and our Dell Command Update application as well.

RobertMiranda
Here to help

Same issue with Windows Update. Outlook and OneDrive not working as well.

GarethR
Meraki Employee
Meraki Employee


This is due to a SNORT definition update seen here: https://snort.org/rule_docs/1-60381
This is a result of a CVE reported by Microsoft seen here: https://msrc.microsoft.com/update-guide/en-us/vulnerability/CVE-2022-35748


The recommendation by Microsoft is to patch the required software. In the meantime you can whitelist the SNORT rule 1-60381 from the Threat Protection centre however this must be understood that you may be vulnerable to the CVE reported.

halotpg
New here

The issue could be related to the ongoing Microsoft incident EX411786. Our current working theory is that Microsoft apps traffic (desktop clients only) is being interpreted as DoS attempts and blocked.

Dunky
Head in the Cloud

Microsoft have acknowledged a service heath issue - EX411786

JessIT1
Building a reputation

Same issues here only our MX84 is affected by these IDS Microsoft Windows IIS denial-of-service attempt

Blocked Outlook on desktops, blocked RDGateway server access, VPN.

 

Setting the MX84 to detection from prevention is a temporary fix. Actions are now being logged as allowed, most destinations are Google Maps, Amazon CDN's, etc.

martin-netx
Getting noticed

 We are also seeing lots of "Microsoft Windows IIS denial-of-service attempt" occurances on multiple MXs. Appears to impact on google drive too if using the app as opposed to browser. 

martin-netx
Getting noticed

Just off the phone with Meraki TAC, engineer was very helpful. They have just now added an IPS rule called OS-Microsoft Windows IIS denial-of-service attempt 1:60381. We should be able to allow this rule if we wish as a form of workaround for the moment. 

Seems that it is a Microsoft update related issue but I'm not too clear on it yet.

JessIT1
Building a reputation

I had our MX84 set to detection temporarily from prevention, are you saying I could put back to prevention then turn on whitelist for Rule ID1-60381 ?  thanks

martin-netx
Getting noticed

Hi JessIT1, I'm just waiting to hear if the IDS allow has worked for actual users...

Wifikohai
Comes here often

Could you explain how to do that?

Thanks

Regards

Rhodri
New here

If you have an MX100 then go to "Security & SD WAN" then "threat protection". Under "Intrusion detection and prevention", add a rule and search for "1:60381". Save changes.

 

You should read the Microsoft CVE and snort rule before doing this, so you can determine whether your environment is actually vulnerable to this exploit.

 

https://msrc.microsoft.com/update-guide/en-us/vulnerability/CVE-2022-35748

https://snort.org/rule_docs/1-60381

TechNick92
Here to help

Reply of Meraki support: IPS/SNORT blocking Microsoft traffic

 

Dear Giacomo,

i can't accept this explanation of Meraki.

There are even problems with big services like Microsoft Exchange Online and so on. So nobody is able to fix this beside Microsoft themselves. 

I understand there is some kind of security issue but it was released hours ago and now breaking traffic for a lot of customers is not the right way to handle it.

Does the whitlisting work or is it dropped as there answers already said after 5-10 minutes? Turning off the IDS is NO solution

Rhodri
New here

We've added an allow rule on our MX100s for this particular type of traffic. It's on the list of rules as "1:60381". This has resolved the problem. We need more information from Microsoft on the potential ongoing threat/mitigation.

Wifikohai
Comes here often

Hi Rhodri,

Please I don´t were to put this rule. Dis you put by youself or you had to call meraki? Could you tell me where is this option? 

Thanks in advance.

Regards

Rhodri
New here
41D5
Getting noticed

What we found:

 

Meraki SDWAN appliance with IPS prevention enabled.

In Security Center, we see this alert:

41D5_0-1660138734890.png

 

Our resolution:

Whitelist SNORT Signature 1:60381 (Click "On" to whitelist)

41D5_1-1660138795822.png

At this point, all of your Office 365 / Internet / Outlook / MS Teams issues should be resolved.  Users should be working.  The users may need to restart apps or reboot.

 

Then patch all Microsoft OS's.  You can't patch until the rule is Whitelisted.

 

After everything is patched, enable the SNORT signature 1:60381 (Click "Off" to remove from whitelist):

 

41D5_2-1660139012167.png

This has worked for 3 organizations where we implemented this fix.

 

 

---UPDATE---

I need to eat a nice plate of crow.  While the fix to whitelist the snort rule works 100%, applying the Windows Updates did not resolve the issue.  When we turn on the SNORT signature, it breaks most clients again.  We thought the Windows Updates fixed it, but it turned out that after some reboots and resets, the applications are still being blocked with the Whitelist disabled.  We also confirmed this in Security Center as we still see incrementing hits on the SNORT rule.

 

So we are leaving the Whitelist ON for now.

 

Rhodri
New here

Is there a patch for Windows 10? The Microsoft CVE only refers to Windows Server operating systems.

AxPayne
New here

I haven't found one as of yet. I've been looking. I checked my machine and only had this cumulative update available. KB5016616

41D5
Getting noticed

To be totally honest, we just ran Windows Update and ensured all of the August 9th patches were applied.  While we thought this fixed the issue with the SNORT signature, it did not.  The client systems may appear to work for some time after the SNORT signature is enabled (Whitelist set to OFF), but the client systems will break after a reboot or after some time.

 

Keep Whitelist ON for now for SNORT Signature 1:60381

BrianMorris
Here to help

I applied this Whitelist ON change in all of our Meraki routers and it seems to have fixed our ability to remotely access their computers and Outlook and OneDrive.

Brandon123s
Conversationalist

The KB patch will not fix this, as the patch only prevents the exploit from working.

 

Blocking of legitimate traffic is simply a false positive. Until they update the rule it will always block legitimate TLS 1.2 handshakes that happen too frequently in accordance to how the rule is implemented.

 

Patch + whitelist is the only method that will work and keep us protected until meraki fixes the rule.

Kevin-C
Conversationalist

Overnight we had 35000 instances of the Microsoft Windows IIS denial-of-service attempt.  The weird part was that we were having lots of internal communication problems.  I could ping devices at a branch, but they couldn't get to some internet sites or internal web sites.  We route all our traffic back though our data centers to be filtered by our firewalls.  The problems only went away when I moved the IDS to detection mode instead of prevention.

Brandon123s
Conversationalist

This is because the exploit mitigation rule targets too frequent TLS 1.2 hellos. Anything using TLS 1.2 could be blocked and some non-microsoft services were at my org according to the logs.

Jamest201
Here to help

Anyone know the KB of the patch? I'm fully patched now and put IDS back to prevention with the rule allowed and everything broke again

DBeiner
Conversationalist
Brandon123s
Conversationalist

Upon applying the rule it took a few minutes for it to take effect for my org. Within 5 minutes the whitelist was recognized and there were no more issues.

 

You get this working?

Rhodri
New here

The patch only applies to servers. It won't prevent Meraki traffic being blocked from "endpoints that are leveraging TLS 1.2", as Microsoft put it.

 

I think we need to wait for Microsoft to close out the incident - MO411804. They're "engaging with their firewall partners" so presumably the snork rule will be corrected at some point.

nlev
Here to help

Latest update on the MO411804 incident:

August 10, 2022 1:56 PM · Quick update
The firewall partner is currently reviewing options to remediate impact.


This quick update is designed to give the latest information on this issue.

TMTECH
Conversationalist

this is the latest alert from MS

 

 

August 10, 2022 10:28 AM · Quick update
We're looking at recent changes made within the Microsoft-managed infrastructure and reviewing endpoints that are leveraging TLS 1.2. Additionally, we're contacting firewall partners to assist our investigation. This quick update is designed to give the latest information on this issue.
 
August 10, 2022 9:53 AM
Title: Some users may be unable to connect to multiple Microsoft 365 services. User Impact: Users may be unable to connect to multiple Microsoft 365 services. More info: Impacted services include, but are not limited to: - Outlook desktop client - OneDrive for Business - Microsoft Teams Current status: After analyzing system telemetry and Fiddler logs from impacted users, we suspect that third-party firewall devices are potentially blocking legitimate Microsoft traffic. Affected customers have reported that disabling firewall rules blocking TLS 1.2 is mitigating impact. We're continuing our investigation into the underlying cause. Scope of impact: At this time, impact appears to be specific to some users who are served through the affected infrastructure. Next update by: Wednesday, August 10, 2022, 11:30 AM (3:30 PM UTC)
haupt
Meraki Employee
Meraki Employee

Meraki Support is aware of the issue. Please view this post Microsoft vulnerability and IPS/SNORT and subscribe to the post for updates. 

 

 

[MOD NOTE: Marking this reply as the solution for greater visibility. Please refer to the Services Notices post linked above (or here: link) for the latest updates from Meraki on this issue]

 

Brandon123s
Conversationalist

I don't agree with the resolution of this issue from Meraki. 

 

Can you confirm that a 100% patched environment does not suffer from the false positive detections? A few people in this thread seem to have stated the false positive detections are still happening despite patching.

 

As I understand it the only way to be protected from this not yet seen in the wild exploit and still have TLS 1.2 working is to whitelist this rule and patch systems.

 

edit: I believe this is now resolved as the affected SNORT rule has been adjusted. At least one post here says this has resolved the issue after re-enabling the rule.

nlev
Here to help

Agreed, I don't think this should be marked as solved just yet.

 

Edit: Sorry, I see it was only marked as solved for visibility, not actually considered solved.

Rhodri
New here

You are correct. Patching does not prevent false positives and the blocking of traffic.

41D5
Getting noticed

@Brandon123s  I totally agree.  The Microsoft vulnerability and IPS/SNORT post makes it sound like the issue is resolved or to call Meraki support.  It should be make clear at this time, that this is an active issue with no resolution.  The workaround is to whitelist the signature.

 

The advisory from Microsoft posted by @TMTECH is much closer to reality.

Rhodri
New here

This isn't a solution. Patching endpoints does not prevent the offending traffic and it doesn't prevent it from being blocked.

 

You cannot currently fix the problem without either disabling IDS or by creating a rule.

SteveBradbury
Conversationalist

Definitely not solved, fix/remove the snort rule please.

JessIT1
Building a reputation

I added whitelist for Rule ID1-60381, can anyone confirm if then putting threat protection back to prevention from detection (a temp fix this morning) will not break connections again..?  thanks

BrianMorris
Here to help

I left my sites on Prevention and just turned the Whitelist setting to ON for this and all of our problems disappeared (OneDrive app, Outlook app, Splashtop, Syncro remote access).Screenshot 2022-08-10 101919.jpg

martin-netx
Getting noticed

We've now got multiple MXs with IDS set to prevention and the whitelisting of Rule ID1-60381. All looks to be working OK now.

 

JessIT1
Building a reputation

After adding whitelist for Rule ID 1-60381 it's showing null not sure if that means it's working.. however I turned IDS back to prevention just as a precaution, did not want to leave on detection, so far desktop Outlook staying connected, VPN connections not dropped.

nlev
Here to help

Same here. It displays null, but everything is working.

Rhodri
New here

Yeah, it says "null" after you apply it but it does work.

Eclipse
Here to help

I don't trust the whitelist because of the odd traffic coming through. We are getting hammered by this address below and I don't want to allow this. Does anyone else see this as well?

 

Eclipse_0-1660143584023.png

 

Brandon123s
Conversationalist

the CVE itself says there is no known exploits for this in the wild. The CVE is from yesterday and it's all about TLS 1.2 hellos.

Anything using TLS 1.2 could be affected here. Tons of companies use amazon AWS, so the above whois probably some vendor hosted service that people authenticate against.

I also have clients in my network connecting to similar aws compute resources and reporting the same thing. The traffic being blocked though is initiated by clients within my LAN, not from the outside. In your security center events log, are you seeing incoming traffic being blocked from that address?

 

BrianMorris
Here to help

Here's the first user that called me this morning -- these are addresses that were blocked with this SNORT rule. Screenshot 2022-08-10 111417.jpg

Eclipse
Here to help

This is what I'm seeing today.

 

Eclipse_0-1660144769492.png

 

Sugarmill_Russ
Conversationalist

This is what I'm seeing.  What concerns me is that the majority are "unknown" affected OS's...Meraki IIS DOS.jpg

Eclipse
Here to help

Meraki isn't great at determining the OS of each client unless you are 100% Meraki gear.