- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
IPS Snort Microsoft Windows IIS denial-of-service attempt - False positive?
Hello,
is anyone else experiencing massive problems connecting Microsoft Windows Outlook to On-Premise Exchange Servers or MS 365 since an hour (round about 11:30 am CEST) ?
All organizations with IPS activated are logging lots of events: Microsoft Windows IIS denial-of-service attempt
Kind regards
Nick
Solved! Go to solution.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi folks! Please be aware that this issue has been resolved as of today (5am GMT August 11, 2022). More info can be found on the Service Notices page here.
If you're still experiencing trouble, please treat it as a different issue and contact Meraki Support. Thank you!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
All of our VPN users, using the client VPN, are having the issues in the UK but onsite people connecting through the MX100s are fine.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Seeing the same thing, there was an option to exclude it from the IDS but it has then disappeared...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We have the same issue. It's affecting all VPN and on-site users. I've logged a ticket with Meraki but haven't had a response yet.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yeah same with us. We have a couple of other systems that are also suffering, pretty much anything that is constantly cloud connected. We fix it briefly by whitelisting or changing a setting but after 10mins it breaks again and our settings on IDS disappear.
On the phone to Meraki support about it, not having a great experience with that at the moment.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
As an option, not one we'd take at the moment, but will turning off the Intrusion Detection and Prevention resolve the issue? Anyone tried?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Haven't tried that but we're seeing thousands of dropped events on our MX100s, which could potentially indicate some kind of attack. So we don't want to disable any security related features.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Disabling and Enabling seems to have stopped it. Not sure if it is temporary.
Edit: lasted five minutes.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Dropping it to detection from prevention does seem to allow a reconnect.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Same for us. Thousands of Events regarding the Snort Rule mentioned in the first Post.
I have switched to Detection instead of Prevention for now till this is fixed. So far "disabling" the IDS or putting it into Detection Mode only seems to fix the Problem.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Switching back to Prevention just breaks things again after 5mins so yeah currently detection mode seems to be the soft fix for now.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes, same problem here causing massive problems for Microsoft desktop applications unable to login using TLS 1.2. We whitelisted the Snort rule (Sid 1-60381) and reported a false positive to snort.org
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
How do you know it's a false-positive?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
-Our own clients access to our own services were affected - only desktop applications, browser access was fine.
-No Endpoint security related issues are reported from our endpoint security stack.
-No issues are reported from the Microsoft security services used.
-Clients that pass through other IPS / Threat detection services than the Meraki / Snort combo report no similar issues.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I think there is some correlation to latest Exchange-server updates, where Microsoft recommended to activate Extended Protection. This changes a lot in the SSL configuation of the IIS. And maybe someone on snort tried to create a IDS rule to block this kind of attack and finally configured some bad preferences to identify a real attack instead of normal traffic to an IIS with SSL
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We're also having issues with Windows Update and our Dell Command Update application as well.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Same issue with Windows Update. Outlook and OneDrive not working as well.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This is due to a SNORT definition update seen here: https://snort.org/rule_docs/1-60381
This is a result of a CVE reported by Microsoft seen here: https://msrc.microsoft.com/update-guide/en-us/vulnerability/CVE-2022-35748
The recommendation by Microsoft is to patch the required software. In the meantime you can whitelist the SNORT rule 1-60381 from the Threat Protection centre however this must be understood that you may be vulnerable to the CVE reported.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The issue could be related to the ongoing Microsoft incident EX411786. Our current working theory is that Microsoft apps traffic (desktop clients only) is being interpreted as DoS attempts and blocked.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Microsoft have acknowledged a service heath issue - EX411786
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Same issues here only our MX84 is affected by these IDS Microsoft Windows IIS denial-of-service attempt
Blocked Outlook on desktops, blocked RDGateway server access, VPN.
Setting the MX84 to detection from prevention is a temporary fix. Actions are now being logged as allowed, most destinations are Google Maps, Amazon CDN's, etc.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We are also seeing lots of "Microsoft Windows IIS denial-of-service attempt" occurances on multiple MXs. Appears to impact on google drive too if using the app as opposed to browser.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Just off the phone with Meraki TAC, engineer was very helpful. They have just now added an IPS rule called OS-Microsoft Windows IIS denial-of-service attempt 1:60381. We should be able to allow this rule if we wish as a form of workaround for the moment.
Seems that it is a Microsoft update related issue but I'm not too clear on it yet.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I had our MX84 set to detection temporarily from prevention, are you saying I could put back to prevention then turn on whitelist for Rule ID1-60381 ? thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi JessIT1, I'm just waiting to hear if the IDS allow has worked for actual users...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Could you explain how to do that?
Thanks
Regards
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
If you have an MX100 then go to "Security & SD WAN" then "threat protection". Under "Intrusion detection and prevention", add a rule and search for "1:60381". Save changes.
You should read the Microsoft CVE and snort rule before doing this, so you can determine whether your environment is actually vulnerable to this exploit.
https://msrc.microsoft.com/update-guide/en-us/vulnerability/CVE-2022-35748
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Reply of Meraki support: IPS/SNORT blocking Microsoft traffic
Dear Giacomo,
i can't accept this explanation of Meraki.
There are even problems with big services like Microsoft Exchange Online and so on. So nobody is able to fix this beside Microsoft themselves.
I understand there is some kind of security issue but it was released hours ago and now breaking traffic for a lot of customers is not the right way to handle it.
Does the whitlisting work or is it dropped as there answers already said after 5-10 minutes? Turning off the IDS is NO solution
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We've added an allow rule on our MX100s for this particular type of traffic. It's on the list of rules as "1:60381". This has resolved the problem. We need more information from Microsoft on the potential ongoing threat/mitigation.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Rhodri,
Please I don´t were to put this rule. Dis you put by youself or you had to call meraki? Could you tell me where is this option?
Thanks in advance.
Regards
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
What we found:
Meraki SDWAN appliance with IPS prevention enabled.
In Security Center, we see this alert:
Our resolution:
Whitelist SNORT Signature 1:60381 (Click "On" to whitelist)
At this point, all of your Office 365 / Internet / Outlook / MS Teams issues should be resolved. Users should be working. The users may need to restart apps or reboot.
Then patch all Microsoft OS's. You can't patch until the rule is Whitelisted.
After everything is patched, enable the SNORT signature 1:60381 (Click "Off" to remove from whitelist):
This has worked for 3 organizations where we implemented this fix.
---UPDATE---
I need to eat a nice plate of crow. While the fix to whitelist the snort rule works 100%, applying the Windows Updates did not resolve the issue. When we turn on the SNORT signature, it breaks most clients again. We thought the Windows Updates fixed it, but it turned out that after some reboots and resets, the applications are still being blocked with the Whitelist disabled. We also confirmed this in Security Center as we still see incrementing hits on the SNORT rule.
So we are leaving the Whitelist ON for now.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Is there a patch for Windows 10? The Microsoft CVE only refers to Windows Server operating systems.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I haven't found one as of yet. I've been looking. I checked my machine and only had this cumulative update available. KB5016616
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
To be totally honest, we just ran Windows Update and ensured all of the August 9th patches were applied. While we thought this fixed the issue with the SNORT signature, it did not. The client systems may appear to work for some time after the SNORT signature is enabled (Whitelist set to OFF), but the client systems will break after a reboot or after some time.
Keep Whitelist ON for now for SNORT Signature 1:60381
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I applied this Whitelist ON change in all of our Meraki routers and it seems to have fixed our ability to remotely access their computers and Outlook and OneDrive.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The KB patch will not fix this, as the patch only prevents the exploit from working.
Blocking of legitimate traffic is simply a false positive. Until they update the rule it will always block legitimate TLS 1.2 handshakes that happen too frequently in accordance to how the rule is implemented.
Patch + whitelist is the only method that will work and keep us protected until meraki fixes the rule.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Overnight we had 35000 instances of the Microsoft Windows IIS denial-of-service attempt. The weird part was that we were having lots of internal communication problems. I could ping devices at a branch, but they couldn't get to some internet sites or internal web sites. We route all our traffic back though our data centers to be filtered by our firewalls. The problems only went away when I moved the IDS to detection mode instead of prevention.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This is because the exploit mitigation rule targets too frequent TLS 1.2 hellos. Anything using TLS 1.2 could be blocked and some non-microsoft services were at my org according to the logs.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Anyone know the KB of the patch? I'm fully patched now and put IDS back to prevention with the rule allowed and everything broke again
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Upon applying the rule it took a few minutes for it to take effect for my org. Within 5 minutes the whitelist was recognized and there were no more issues.
You get this working?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The patch only applies to servers. It won't prevent Meraki traffic being blocked from "endpoints that are leveraging TLS 1.2", as Microsoft put it.
I think we need to wait for Microsoft to close out the incident - MO411804. They're "engaging with their firewall partners" so presumably the snork rule will be corrected at some point.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Latest update on the MO411804 incident:
August 10, 2022 1:56 PM · Quick update
The firewall partner is currently reviewing options to remediate impact.
This quick update is designed to give the latest information on this issue.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
this is the latest alert from MS
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Meraki Support is aware of the issue. Please view this post Microsoft vulnerability and IPS/SNORT and subscribe to the post for updates.
[MOD NOTE: Marking this reply as the solution for greater visibility. Please refer to the Services Notices post linked above (or here: link) for the latest updates from Meraki on this issue]
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I don't agree with the resolution of this issue from Meraki.
Can you confirm that a 100% patched environment does not suffer from the false positive detections? A few people in this thread seem to have stated the false positive detections are still happening despite patching.
As I understand it the only way to be protected from this not yet seen in the wild exploit and still have TLS 1.2 working is to whitelist this rule and patch systems.
edit: I believe this is now resolved as the affected SNORT rule has been adjusted. At least one post here says this has resolved the issue after re-enabling the rule.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Agreed, I don't think this should be marked as solved just yet.
Edit: Sorry, I see it was only marked as solved for visibility, not actually considered solved.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You are correct. Patching does not prevent false positives and the blocking of traffic.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Brandon123s I totally agree. The Microsoft vulnerability and IPS/SNORT post makes it sound like the issue is resolved or to call Meraki support. It should be make clear at this time, that this is an active issue with no resolution. The workaround is to whitelist the signature.
The advisory from Microsoft posted by @TMTECH is much closer to reality.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This isn't a solution. Patching endpoints does not prevent the offending traffic and it doesn't prevent it from being blocked.
You cannot currently fix the problem without either disabling IDS or by creating a rule.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Definitely not solved, fix/remove the snort rule please.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I added whitelist for Rule ID1-60381, can anyone confirm if then putting threat protection back to prevention from detection (a temp fix this morning) will not break connections again..? thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I left my sites on Prevention and just turned the Whitelist setting to ON for this and all of our problems disappeared (OneDrive app, Outlook app, Splashtop, Syncro remote access).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We've now got multiple MXs with IDS set to prevention and the whitelisting of Rule ID1-60381. All looks to be working OK now.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
After adding whitelist for Rule ID 1-60381 it's showing null not sure if that means it's working.. however I turned IDS back to prevention just as a precaution, did not want to leave on detection, so far desktop Outlook staying connected, VPN connections not dropped.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Same here. It displays null, but everything is working.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yeah, it says "null" after you apply it but it does work.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I don't trust the whitelist because of the odd traffic coming through. We are getting hammered by this address below and I don't want to allow this. Does anyone else see this as well?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
the CVE itself says there is no known exploits for this in the wild. The CVE is from yesterday and it's all about TLS 1.2 hellos.
Anything using TLS 1.2 could be affected here. Tons of companies use amazon AWS, so the above whois probably some vendor hosted service that people authenticate against.
I also have clients in my network connecting to similar aws compute resources and reporting the same thing. The traffic being blocked though is initiated by clients within my LAN, not from the outside. In your security center events log, are you seeing incoming traffic being blocked from that address?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Here's the first user that called me this morning -- these are addresses that were blocked with this SNORT rule.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This is what I'm seeing today.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This is what I'm seeing. What concerns me is that the majority are "unknown" affected OS's...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Meraki isn't great at determining the OS of each client unless you are 100% Meraki gear.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes. That address is where the majority of the DoD events are coming from which is triggering the issue on our end. I've never seen this much traffic from AWS before and was wondering if anyone has seen something similar. I have emailed their abuse center for what it's worth.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
So no patch for Windows 10?
Rule with Prevention is still blocking for us.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Same. Rule is set to "whitelist" but shows as null. Others say it is working, but still showing blocked events for us.
Found this helpful? Give me some Kudos! (click on the little up-arrow below) and If my reply solved your issue, please mark it as a solution 🙂
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
What device do you have?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
MX 100 at our primary site.
Found this helpful? Give me some Kudos! (click on the little up-arrow below) and If my reply solved your issue, please mark it as a solution 🙂
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Not sure if it helps but ours is set like this and it works.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes. That still shows blocking in Events.
MX250 Current FW 17.7
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Experiencing the same on our MX84. Had to switch from prevent to detect and working this way right now.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We have set IDS to detect and got the null: 160381 whitelist rule in place but Microsoft VPN still appears to be blocked - does our MX84 need restarting or power cycling?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Is anyone else getting a ton of "Microsoft wimgapi LoadIntegrityInfo heap buffer overflow attempt" blocks after allowing "Microsoft Windows IIS denial-of-service attempt" on their MX?
Seems off.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The latest from Microsoft:
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We have set IDS to detect from prevent and got the null: 160381 whitelist rule in place but Microsoft VPN still appears to be blocked - does our MX84 need restarting or power cycling?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Here's what mine looks like. Make sure that you apply it to all sites if you are adding manually. I took the easy route and went to Security Center, it shows all sites and I just clicked on "Microsoft Windows IIS denial-of-service attempt" and clicked Whitelist ON.
No reboots needed!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes that's what we have set on MX84 - all devices onsite now able to connect outbound to Microsoft services e.g. outlook to Exchange online, etc
But when trying to connect to Microsoft RDS VPN system not connecting from offsite
Puzzled
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Perhaps your Microsoft RDS VPN issue is unrelated to this IDS rule. Do you see any other rules in the security center relating to the impacted clients?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
It's really disappointing that Meraki didn't push out a fix when the details of this first emerged. Instead, each customer whose email was nonfunctional was left to figure out the problem and manually add a threat protection override. What's the point of something being in the cloud if the vendor can't act fast when there is an obvious and serious problem?
The irony is that the vulnerability allows someone to conduct a denial of service attack, and the result of Meraki protecting from it is a denial of service.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I only added the allow list entry to rule to a couple networks but now see it on more. Did Meraki add that entry themselves as a work around while things get worked out between MS, SNORT and Meraki?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
No, we saw the same thing and others have been reporting that as well. Adding it to one network seems to unexpectedly add it to all networks in the organization.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
yes we were on to meraki support this morning - they didn't mention this issue, they told us we had hardware issue and to swap in our backup MX84 - very poor
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 out 6 of our devices were stopping traffic due to SNORT 1-60381. Cannot get any answers as to why whitelist a working rule. Now the rule has been deleted from SNORT all together? What was found to be wrong with the rule?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The official update says: "The Snort rules have been removed to reduce the impact. " Was there another rule other than 1-60381 that was contributing to the issues?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am still seeing traffic blocked by 1-60381 SNORT on my MX100
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
difficult to tell if it was us changing and rebooting MX and RDS gateway server or if MX84 received Meraki update removing that IDS snort rule - but we do seem stable now and MS VPN is working again.... a whole day of pain that I think Meraki could have been quicker and better at communicating to its customers
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
What Meraki have not communicated is if we can change IDS back from detect to prevent - has Meraki made a statement?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The offending SNORT rule has been removed and Meraki stated things should be fully resolved by 3PM PDT. So I'd say the conservative approach would be to wait 30 minutes then switch IDS to prevent. I allow-listed the offending rule and stayed on prevent - for me this resolved the issue almost immediately.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I wish Meraki would have posted the information on the dashboard early this morning.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Is it true if your Windows servers and Windows 10 PCs were on the latest OS level AND had the patch from 8/9 the traffic would NOT have been blocked?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
No. Meraki was blocking traffic regardless.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
So at this point with my MX84 threat protection set back to prevention and Meraki removing the Snort rule, should we remove the whitelist rule for 1:60381 in our Intrusion detection and prevention?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Anybody else having new problems today? We added the Allow rule for 1:60381 to our Threat Protection page to clear up the issues, and that rule is still there today. But...about 2 hours ago, we started having all sorts or problems again. We were unable to visit some webpages. We had back-end processes on servers that communicate with Web APIs that no longer work.
Thinking of how we had bizarre problems yesterday, I set our IPS mode to DETECTION and all of our problems went away a few minutes later. So clearly the IPS in the MX100 was dropping traffic. But I haven't been able to determine anything about the IPS rule. When I search our Events list for all IPS items, I get an empty list returns. It seems crazy to me that we have no events for IPS, not even a Rule Update event.
Anyone else out there having new problems with the IPS today?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Nope. No reports, no indications of anything similar to yesterday where all sites were affected and IPS events during the time frame increased from a normal of a few hundred to tens of thousands.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
No issues here, Still have the rule whitelisted (will remove next week). No unusual IDS events showing in Security Center.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Same, no further problems, but like you left the whitelist one in, to change at later date
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
In Security Center, we've gotten a few of these events:
SERVER-OTHER Exim unauthenticated remote code execution attempt
Rule ID1-53376
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi folks! Please be aware that this issue has been resolved as of today (5am GMT August 11, 2022). More info can be found on the Service Notices page here.
If you're still experiencing trouble, please treat it as a different issue and contact Meraki Support. Thank you!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
How do we go about getting a root cause analysis for this? I appreciate that it has been resolved, but there are certainly some unanswered questions. A few I can think of:
- Why did this rule seem to impact more traffic than it was intended to?
- How can we review the specific behavior(s) that the rule was matching on?
- What's the review process for new IPS/IDS rules being added?
- How can customers be aware when a new rule is being added?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @billyzoellers, those are great questions. Currently, this is under investigation still. Once we have more information to share, we will update everyone. Stay tuned!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I see that if we follow the link to the rule details in Snort this rule now shows as "deleted". I guess this means we can switch off the whitelisting of this rule if it has actually been deleted?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am removing our whitelisted rule tomorrow, I don't expect any issues.