Event 10036, DistributedCOM

[BM403]

Hello,

Have an error that just started occurring last Tuesday, September 14th after updating my domain controllers. I'm getting the following error; 

The server-side authentication level policy does not allow the user domain\user SID (X-X-X-XX-XXXXXXXXXX-XXXXXXXXXX-XXXXXXXXX-XXXXX) from address 10.0.100.254 to activate DCOM server. Please raise the activation authentication level at least to RPC_C_AUTHN_LEVEL_PKT_INTEGRITY in client application.

The IP address (10.0.100.254) is my MX IP Address. This error I believe has to do with Active Directory Authentication I have set up under Security & SD-WAN > Active Directory. I'm getting the same error for all 3 sites each pointing to the IP address of the MX device. The error occurs roughly every 2 hours in my event log on my domain controllers.

In my test network, I disabled Active Directory Authentication on that MX and the errors stopped. I have googled my heart out for Event 10036, DistributedCOM, and can't find anything related to the above error. I did find one someone on Reddit who experienced the same issue but it was on a Palo Alto firewall. 

Does anyone else have this happen since last week's patch Tuesday? I'm not even sure how I would go about raising the activation authentication level at least to RPC_C_AUTHN_LEVEL_PKT_INTEGRITY in client application.

Everything works fine and users can still login into the client VPN using their AD credentials. But, this error is really strange and is filling up my event logs. 

When I run a dcdiag everything passes, but the error shows up under SystemLog. 

Could this be a possible bug? 

[PaulStanley]

I just noticed this error this morning while looking for another issue. 

[NetBeast33]

Having the same issue. I raised the authentication level to Packet Integrity on a Server 2016 DC, rebooted and errors still present.

Checked my MX device and had a generic WMI error that referenced setting up MX device to authenticate to AD.

Configuring Active Directory with MX Security Appliances - Cisco Meraki

Read the link a bit and went back to the device and error is no longer present.

Looking into it further tomorrow. Will update if any progress.

[BM403]

I get those WMI errors on my MX every so often. I believe this is on MS to resolve and hopefully address the issue soon. No impact to my AD authentication because of the error, it's just really annoying seeing it fill up my event log. 

[ALibrarian]

Hi! I've seen this error attributed to the September update - KB5005568: 

https://docs.microsoft.com/en-us/answers/questions/564347/server-2019-update-kb5005568-sept-2021-for... 

[BM403]

Thanks for this link, ALibraian. I will continue to monitor that post and see if anyone comes up with a solution. I've tried a number of fixes for this and have come up empty. 

[s-i]

Hello,

Do you have any update for this ? We have the same problem.

Best regards

[NetBeast33]

I have not found a solution other then uninstalling the Windows update.

https://community.spiceworks.com/topic/2332445-event-10036-distributedcom-error-on-both-dcs?page=1#e... 

[s-i]

Hello,

It's not a solution.

Meraki doesn't do anything, it's incredible.

Best regards

[FXFinVT]

Same here. This is on Meraki and Microsoft to work out.

[BM403]

I would have to agree. I’ve tried numerous work arounds to try and get the error to stop and turning off AD authentication is not the answer. This has been going on for months now and other than clogging up my event logs on my DCs I still haven’t noticed any negative impact. 

[NBP]

Any update on this issue or is Meraki just pretending it doesn't exist?

[JCooper]

Have this same issue and resolve?

[Robthesoundguy]

Following this as we've been experiencing this issue since deploying our Windows Server 2022 DC's. Hopefully Meraki engineering eventually figures this out. 

[Kazuokei]

We are in 5-4-2022 and still Meraki not found a way to fix it, poor support as usual.... we are masochists using those pieces of plastic....

Other brands have solutions for it:

https://supportportal.juniper.net/s/article/SRX-Integrated-User-Firewall-with-Active-Directory-stops...

https://www.sonicwall.com/support/knowledge-base/troubleshooting-single-sign-on-sso-related-errors/1...

https://live.paloaltonetworks.com/t5/general-topics/i-am-having-pan-os-integarted-user-id/td-p/43968...

[FSaucedo]

I have fixed this issue in our setup. Same error. Turns out that the Active Directory authentication done by the MX box requires TLS. For that, the chatter uses an SSL certificate. I had just finished replacing our DC with a newer box. Since it was only being used as a DNS server, only the barebones installation was done. As you, I panicked a bit when I saw the errors fill up the event log. After sleuthing around and reading posts everywhere, ya'lls reminded me about the active directory link with the MX box. Checking on it, the console reported the error where it could not authenticate with the new DC. Reading the requirements for this setup, I found that it needed an SSL certificate. I installed the IIS role and created a self-signed cert. Once that was done, the MX was able to connect and when I checked the server's event log, the messages no longer appeared. Hope this helps! 

[Kazuokei]

Thanks for your feedback. It is an option and also setup AD as Certificate Manager. Is really good to know we are solving this becasue if we wait for Meraki support  it wil takes ages....

[NBP]

My DCs have certs that are automatically issued from my internal Windows Server CA, but I think the subject name is the DNS hostname of my DC, and there are no IP address SANs on the certs. In order for the certs to work correctly, I think the Meraki needs to connect to the DC's via hostname and not IP address? I'm wondering if that's my issue? If I replace the IP address with the DNS name in the Meraki admin portal, it gives me an error that it's an invalid server IP. I also am not seeing where I can specify an internal DNS server in the Meraki admin portal so it can resolve the DNS name. Am I missing something?

[Kazuokei]

I connected them with IP address. The key is create an local user on AD server with WMI read only options. Let me atach here the steps that I followed:

1. Create and account in AD enrolled in Domain Users and Account Operator domain groups. In this case account is: usrmeraki (this an example name)

2. Setup AD CS as follow: https://dinika-15.medium.com/installing-active-directory-certificate-services-ad-cs-4db7d0950289

3. Create a certificate for Domain Server to permit Client Authentication and Server Authentication opening manage Computer Certificates: certlm (run comand in CLI as administrator)

4. Expand Personal and over Certificates, right clic and request a new certificate, follow the wizard and check

Domain Controller Authentication and then click on Enroll.

5. Validate new certiifcate is created in: certlm (run comand in CLI as administrator)

6.  Grant WMI acces under root\cimv2 usrmeraki account in AD Server as follow

WMI Control --> Security (tab) --> CIMV2 (tehn click on security button)

Add user usrmeraki and enable: Enable Account and remote Enable

7. Grand Permission over DCOM components AD Server as follow: dcomcnfg (run comand in CLI as administrator) 

Right Click on My Computer (Left Panel) selct propertties. Go to COM Security (tab)

In Access permision: add usrmeraki with Local Access and Remote Access

In Launch and Activation Permissions: add usrmeraki with Local Access and Remote Access

8. Go to Open Meraki web console and test credentials under active directory menu  and test conectity and read groups/users you want to include in MX device. Fill boxes as follow:

Short domain: yourdomain.com

Server IP: IP Address for AD Server

Domain admin: YOURDOMAIN\usrmeraki

Password: Your Password

9. With this conifugration you will see green check on status and integration is working as expected.

Hope this helps you 🙂

[walner_borbon]

Thanks for the info!! For me, the pending thing was point number 4! Because we already have CA

[Kazuokei]

Do you have look?

[Fingers]

All -

The 16.16.3 firmware resolves the 10036 DCOM  issue.

[Kazuokei]

Hi, this update don't fixed it. Trust me.

[BM403]

Meraki has not released an update to resolve this. The only solution for me was to disable the authentication level of RPC_C_AUTHN_LEVEL_PKT_INTEGRITY or higher for activation.

This method will only work until March 14, 2023. Then Microsoft will enforce it with no way to disable it.

You can find the article here;

KB5004442—Manage changes for Windows DCOM Server Security Feature Bypass (CVE-2021-26414) 

[Kazuokei]

Is the same, it not works, believe me. The only thing worked for me, I published above.

[milenz]

16.16.3 firmware reduces the number of DCOM errors in the logs but does not resolve the issue. I spoke with Meraki support this week, and they acknowledged this as a known issue. The devs are supposedly working on this. In the meantime, their recommendation was to set the reg key as per the MS article mentioned earlier.

KB5004442—Manage changes for Windows DCOM Server Security Feature Bypass (CVE-2021-26414) (microsoft...

[Kazuokei]

I'm on MX 17.8... and nothing changes, from my view.

[Lloydy]

almost a year later and this is still an issue ..... ?

please tell me someone has a fix for this.

[Kazuokei]

Hi, try with comment above on date: ‎06-22-2022 12:50 PM . Hope it can helps.

[ErinT1]

This continues to be an issue, we cant upgrade our domain controllers to server 2022 because the regfix that microsoft recommends doesnt work on server 2022. We are also now starting to get sporadic issues with the firewalls not picking up the correct user information as well as high CPU usage on our DC's. Meraki support don't seem to have any idea when the developers are actually going to fix this, and ive had a ticket in about it for 6 months. astoundingly poor considering if it doesn't work then our entire capability to safeguard staff and students doesn't work as we rely on applying policies based on staff/student group memberships and then applying to devices based on who is logged in.

What's even worse is that this area of policy application even when working doesn't actually show in the client list of devices, only in event logs, which is frankly the dumbest design decision I have ever seen for what is generally a really well designed dashboard. I need to be able to see what policy is applied to a device from the client list not trawl through event logs to see.

[Jedediah_Jumbl]

Been seeing the same thing since July of this year, and have had a case open since then.

Was most recently advised to update to the stable beta candidate of 17.10.1, problem persists.

Something I did notice was that for a 2016 DC that I have, there were "connected to domain controller" event logs that started populating within Meraki after updating the associated network to 16.16.5. However, it did not change the fact that there are still persistent event logs showing up on the DC itself, nor the fact that within the Dashboard, the DC in question still shows a "WMI error" status.
On a related note, I also have a 2022 DC that is in the same network as the 2016 DC, and after the upgrade to the 16.16.5 firmware, it was still spamming "cannot connect to Domain Controller" events in Meraki, as well as the "server-side authentication level policy" / "RPC_C_AUTHN_LEVEL_PKT_INTEGRITY" messages on the DC itself.

Anyone else having any luck ?

[ErinT1]

I had a beta firmware uploaded onto one of our sites to test from the developers. Unfortunately no help so far, info had been passed back to the developers to continue to work on.

[Kazuokei]

Same speech since pandemic started......

[Jedediah_Jumbl]

If you're referring to your post from 6-22-2022, I did actually try that, as Meraki had recommended something similar, while simultaneously referencing their https://documentation.meraki.com/MX/Content_Filtering_and_Threat_Protection/Configuring_Active_Direc... article and suggesting that it was something mis-configured on our DCs.

It did not change anything in terms of the behavior, and Meraki has acknowledged months ago (at least to me on my case) that this was a known issue related to their firmware that they are still working on.

[Kazuokei]

Yeah, when I said same speech before it is about Meraki Staff.

[chrisc543]

Same issue here. Anyone that has ticket open hear anything? Assuming not as Meraki support never replies to known issues. 

[Kazuokei]

Never replied..... 

[Robthesoundguy]

I had email dialogue with support back in November that seemed promising, but then they replied back and basically said "it's with the developers now" I just re-opened the ticket to see if there's an update available, but I'm not optimistic. 

[ErinT1]

18.104 I was told fixes it. It doesn't. It appears to rectify the DCOM errors in event viewer on the DC’s but doesn't pull any user information so is essentially as broken as ever, I continue to chase but at 9 months on the fact its not fixed tells me the developers don't consider it the HUGE problem that it is.

[ErinT1]

Further testing with firmware version 18.104 to me suggests it does rectify the DCOM aspect of this, but the WMI query that the server passes can easily break and then break the entire AD integration facility. The below i believe is the reason for it. they are using the IWbemServices::ExecQuery method
In their code which if the data set is too large or the DC doesn't provide the information in a specific time means it just breaks the whole query. They need to be using the IEnumWbemClassObject::Next method.

The below is an example of an error i've seen that shows the memory max being hit, i also receive others but they relate to errorcode 0x80041032, which then when looking at the article further below from microsoft explains the potential error and how to fix it. Which would require the Meraki coders to amend their code to use the "Next method".

Id = {00000000-0000-0000-0000-000000000000}; ClientMachine = ; User = nnn\Ouradmin; ClientProcessId = 2836; Component = Unknown; Operation = Start IWbemServices::ExecQuery - root\cimv2 : SELECT EventCode,InsertionStrings,RecordNumber FROM Win32_NTLogEvent WHERE Logfile = 'Security' AND EventType=4 AND (EventCode=540 OR EventCode=672 OR EventCode=4624 OR EventCode=4768) AND RecordNumber > 4100256909; ResultCode = 0x80041032; PossibleCause = Throttling Idle/stack Tasks in hitting Max Memory quota

https://learn.microsoft.com/en-us/troubleshoot/windows-client/system-management-components/wmi-activ...

I think there are other issues as well as the above, as i cant get any Domain Controllers on server 2022 to pull user information at all despite them showing green on firmware 18.104. This could also be the above issue but im unable to tell for sure there isnt another issue influencing it.

[ErinT1]

I stand corrected, on 18.104 still appear to get DCOM hardening errors on some servers so doesn't appear to rectify the issue. Which makes sense as the notes on the firmware speak about using newer DCOM protocol version but nothing about upping the protocol security level.

[Jedediah_Jumbl]

Here we all are, over a year into many of us having this issue, and Microsoft is going to begin enforcing this in March - DCOM changes first released in June of 2021 become enforced. See https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2021-26414 and https://support.microsoft.com/en-us/topic/kb5004442-manage-changes-for-windows-dcom-server-security-....

Are we all just basically screwed if they don’t get this figured out, since there will be no way to disable it, and since we are all likely past the point of using a firmware version that did work?

I cannot fathom how they haven’t worked this out

[Jedediah_Jumbl]

So I had gotten an update from Meraki with the case I've had open that there was a permanent fix as part of the current stable release candidate firmware, ver. 18.105.

I updated one of my smaller networks to it last night, which is running an MX64, and to my surprise,  there are no longer spammy event logs in the Meraki dashboard for that network's "Security Appliance" section as there have been. Additionally, none of the DCs that I have configured in the "Active Directory" tab for that network are showing a "WMI error message", but shockingly are showing "Success". Finally, I'm not seeing spammy logs on any of those DCs detailing the DistributedCOM source, nor the associated "The server-side authentication level policy does not allow the user <User> <SID> from address <address> to activate DCOM server. Please raise the activation authentication level at least to RPC_C_AUTHN_LEVEL_PKT_INTEGRITY in client application."

I have yet to test this to confirm that it is actually the fix, but if anyone else has time to weigh in if they get a chance to test it, I think many people would be most appreciative.

Meraki may have nipped this in the nick of time.

[FSaucedo]

@Jedediah_Jumbl 

Can confirm that v18.105 shows a green status in the Active Directory authentication. Using an MX100. Doing a happy dance....

[Robthesoundguy]

I've installed the SRC firmware on two different orgs. One with a MX68 and another with a MX95 and can confirm that this resolved the issue. I now have green check marks, the 'unable to connect to domain controller' entries in the logs and the server-side log entries have disappeared. 

While I'm happy that the issue is resolved, I am a bit disheartened that it took Meraki this long to fix it. 

[ErinT1]

Just so you and others are aware that having green ticks in meraki and no errors on the DC doesnt necessarily mean its working. Are users being picked up in event logs area and are group policy based rules hitting those users correctly as per whatever setup you have? Doesnt appear to be universally fixed in my environment with all green etc, still have broken MX boxes not pulling user data and applying group policies as they should.