MX100 AMP Blocking Microsoft Update and Java Update

[Jack]

We bought 2X MX100 Security Appliance (retail price at $4999 each + License ). Currently running at the latest Stable firmware 12.24 and It blocks all device from downloading windows update and Adobe update even thou I whitelist all known Microsoft update sites.  Meraki solution

1) Disable Amp ( Risk of getting Malware )

2) Upgrade firmware to V14 BETA. ( Running critical production network on BETA Firmware? )

Anyone have better workaround please help !

[PhilipDAth]

I've deployed a lot of MX's - and they have never blocked Windows Updates without being configured to do so.

Have you configured are layer 7 firewall rules?  Can you configured any content filtering rules?

[Jack]

Hi PhilipDath,

L7 only block All P2P, Video and Music and Gaming.

Content Filtering only blocks some category that has nothing to do with Microsoft update and Adobe update. ( Unless Microsoft uses P2P Protocol to push update? ) 

This only happens on my MX100. I have many MX65W with the same config without any issue. 

I called Meraki support twice regarding this issue for a month now and they gve me the same answer. 

Hope to hear back from some other MX100 users.

[PhilipDAth]

Try removing the L7 rules and see if that fixes it.  If not put them back.  Repeat with the contenting filtering rules.

One of those items should get it working again. Tell us which one it was.

[Ryan-Zimmerle]

Hello @Jack

I have an MX100 sitting as a cold spare to our MX250.  I will fire this up and create a test network and try to duplicate the issue.  When we had the MX100 in operation AMP was grabbing Console8 updates as malicious.  I am assuming AMP is enabled and what are you IDS settings? Prevention and Balanced?  Just want to duplicate your settings here.  

[Jack]

Thanks for your help. IDS set to Prevention - balance.

[Ryan-Zimmerle]

Hello Again @Jack

I have a spare MX100 running 12.24 that I reset back to factory and I enabled AMP and IDS like you have, see screenshots.  I also added the L7 rules you mentioned above.  I happen to have an extra connection to the outside world with a public IP, so there is not a double NAT taking place here.  I had no problem fetching updates from windows update servers or adobe updates.  if this traffic was getting grabbed by IDS or by AMP, there would be a log of that event that is easy to find the in security center.  

This very much sound like an issue with Content Filtering, more specifically IP/URL reputation as @PhilipDAth mentioned.  

"In firmware version 13.3, URL reputation was prioritized over IP reputation, as opposed to IP reputation being the deciding factor on previous firmware versions. If, for some reason, the IP has a different categorization then the URL, the client could be allowed through."

I can tell you that I am running MX 14.15 on an MX250 and I have not been adversely affected by this beta firmware in a production environment with 1000+ daily clients.  

"If a client is being blocked from accessing a page, the easiest way to tell whether content filtering is blocking the traffic is to check your Event Log. When looking at the Security Appliance's network in the dashboard, navigate to Network-wide > Monitor > Event log. To help narrow down the scope, the event type 'Content filtering blocked URL' can be included in the 'Event type include' field."

I hope this helps. 

Ryan 

[PhilipDAth]

If you look in the event log for the network - what is the exact reason it gives for the blocking the traffic?

[Jack]

I dont think Event log shows whats being block on AMP. Any Idea what event to sort?

[PhilipDAth]

If you go:

Security Appliance/Security Centre/Events

Does anything come up?

[Jack]

I don't see anything special within that log. We have 100+ PC and all 100+ Pc cannot update windows when AMP is turn on . Meraki support solution is to update to the beta firmware which im not very comfortable doing... just looking for a workaround for now.

[PhilipDAth]

And you are saying that if you disable AMP it starts working?  If there is nothing in that log then it should mean that AMP is not blocking your traffic.

The beta firmware is pretty good.  You are unlikely to find any issues if you upgrade to it.

[Jack]

Yes as soon as i disable AMP everything work. When i turn it on then Windows update stop working again. I think is a knows MX100 issue.

Here is what show up on the log but i dont think those are windows update

Oct 13 12:19:22 IDS Alert 209.66.87.99.IPYX-073920-004-ZYO.zip.zayo.com
209.66.87.99:80

S
Blocked
BROWSER-IEMicrosoft Edge xlink type confusion memory corruption attempt
Oct 13 12:09:00 IDS Alert a23-219-162-115.deploy.static.akamaitechnologies.com
23.219.162.115:80

S
Blocked
BROWSER-IEMicrosoft Edge xlink type confusion memory corruption attempt
Oct 13 12:04:00 IDS Alert a23-219-162-115.deploy.static.akamaitechnologies.com
23.219.162.115:80

S
Blocked
BROWSER-IEMicrosoft Edge xlink type confusion memory corruption attempt

[PhilipDAth]

We can probably solve this now we know the IDS is triggering.

Go:

Security Appliance/Threat Protection/Intrusion detection and prevention

Under "Whitelisted Rules" click "Whitelist an IDS rule".  Select the rule that is firing above the in the log.

[Ryan-Zimmerle]

@Jack  May I get a screen capture of your content filtering and layer 3 / 7 rules? 

You are running MX 12.24 correct? 

Ryan 

[Jack]

yes 12.24

Sorry i cannot share my L3/7 rule here. preety basic..only blocking few /32 ip. CF blocking P2P, Video and Gaming

[PhilipDAth]

I think you might be affected by the IP Reputation/URL filtering issue.  This was resolved in 13.3.  I think you should upgrade to the beta firmware.

You can read about the issue here:

https://documentation.meraki.com/MX-Z/Content_Filtering_and_Threat_Protection/Content_Filtering/Cont...

"Sometimes, sites will be blocked even though their URL category is not blocked. Usually this happens when the IP has a bad reputation but the URL reputation is good. This happens commonly with very large domains like Google that own many IP addresses and sometimes purchase new IP addresses that have not yet been re-categorized to take their new owner into consideration. In situations like this, these IPs sometimes have a category of 'Phishing and Other Frauds,' or various other categories that may actually be blocked:"

[Jack]

Meraki support told me that V13 will not even solve my issue. I have to schedule a firmware update and they need to manually push V14 for this issue to be resolve. But its on Bata. Scary. I just dont understand why Cisco Meraki cannot make a windows update to work on a stable firmware? 

[ARiK_LeV]

I would assign a group policy  only to the server  to disable  AMP  just for those devices.  Then try windows update again.   

[PT]

Seen  similar problem with MX64/65/84.

Found that it corrected by turning AMP off, waiting a bit (minutes) then turning it back on, this allowed updates to proceed.

Havent seen the problem in a while, so may have been covered in a recent update - we are running typically newer than stable release.

[Ryan-Zimmerle]

@Jack  I wrote out a really lengthy reply and added screenshots, it now disappeared or was removed, did you get a chance to see that reply? 

[Jack]

I saw it on my email and Im trying to reply and then its gone on the forum .. someone deleted it maybe for privacy issue? Its funny that it works for you but not me.  I did not have the chance to look at your screenshot. I guess i have no choice but to upgrade to the new beta firmware... im sure it will work. Worst case Revert back to V12. Thank you again for all your help !! 

[Ryan-Zimmerle · Accepted solution]

@Jack

I am not sure what it was removed, there was nothing in there that was a privacy concern.  Anyway, earlier I was testing with a Win 7 box, when I tested with a Win 10 box, bam right away Windows Update broke.  I am running MX 12.24 on this MX 100, I moved the client over to my MX 250 running MX 14.XX and right away the updates started working.  I can confirm there is an issue here and I was able to replicate it exactly as you described.  

Ryan

[Jack]

Yes its a known issue i guess... just not happy with Meraki ignoring this major issue for so long... Its been going on for months... their only solution is by upgrading to the Beta firmware. Why is there not a stable version that has this issue resolved? Why does paid customer production network needs to be their lab rat? Anyway ill update to V14. Thank you for doing all this for me !!

[chrismoses]

This is STILL and issue (11/2018). We haven't been able to access any Windows updates for over a month. We did have AMP enabled at our colo (MX 100 running 13.33) How TF can this be an open issue with production firmware Meraki? Not being able to update Windows is almost criminal. Disabled AMP on the network and updates are working.

[NolanHerring]

@chrismoses

I understand the frustration, however I think it might be OK now to upgrade to 14.X if your willing. That seems to have fixed the issue based on others from this thread.

[chrismoses]

This will be the third *major* issue that we've encountered this year where the fix was installing beta firmware. That's nuts.

[Jack]

Disabling AMP for 10 min and enable it works for me. Try that. 

[CarolineS]

Hi @Jack and @Ryan-Zimmerle -
Apologies about the message getting deleted - our community's spam filter is a bit over-active and it flagged that post. I added it back. I'm also looking into how to tone down the filter!
Cheers!
- Caroline

[Ryan-Zimmerle]

Hello @CarolineS, 

Thank you for jumping in here and letting us know, so nice to have some Cisco Meraki presence here.  

[Shanec]

We to have issues with AMP on our MX100 for various downloads saying its network error etc. Not willing to go on betas on a production environment and i know turning off AMP will sort it but defeats the purpose of paying for the advanced licence.

[Jack]

We have the same issue. This seems to be a known problem with the MX100 only. I tried everything to find a workaround but no luck. Here is basically your option.

1) turn off AMP
2) upgrade to v13 firmware beta ( I did that on my environment with 500 users and multiple vpn etc and it works great so far.) worst case 1 click roll back to V12. I understand the word beta is scary but v13 already been around for a long time. V14 already available but Meraki support need to manually push it on their end. My suggestion is upgrade to V13 and keep your eye on it.

[Sbadza]

I'v been having same issue since 2016. A couple of our Meraki sites (MX64's) have reported file download failures when AMP is enabled. This issue manifest itself in a weird way, they work sometimes.

 https://documentation.meraki.com/MX-Z/Content_Filtering_and_Threat_Protection/Advanced_Malware_Prote...)

This is a known issue with Cisco Meraki AMP, Sometimes files will change disposition based on new threat intelligence gained by the AMP cloud and sees clean files as Malicious, then blocked.

Per Meraki, most customers are experiencing similar issues and they are working on a permanent fix soon????

Since we don’t want to disabled AMP as a fix, here is a workaround;

1. Turning AMP off & on for 10 minutes and then whitelist the URL sometimes seems to do the trick.
2. In some instances, code upgrade to 13.25 beta firmware may resolve the issue, but I won’t recommend this, as they have not always worked for every MX and can cause other network issues (Verify with Meraki first).

[c0sm0]

Yup seen this too. Meraki MX64 and 64W.

Solution is to add site to whitelist, turn off AMP - wait, turn on AMP - wait.

I whitelisted the following for Windows Updates..

microsoft.com

windowsupdate.com

Meraki filtering assumes all subdomains allowed as well on the above.

[mw_awa]

Thank you, just wanted to reply stating c0sm0's workaround fixes the issue. I'm running WSUS for domain joined machines, but some BYOD laptops on our wifi could not get windows updates over the internet. Running MX100 and MX64 on 13.3

[StevenBosco]

I just had a similar issue, and wanted to describe it for others' reference. All Windows 8 era machines (8, 8.1, WS2012, WS2012 R2) would not update and gave the error code 0x8024402F. This began seemingly sporadically in November of 2018, and audit logs did not show any system configuration changes around that time. The ultimate cause appeared to be AMP blocking Windows from downloading legitimate .cab files from Microsoft websites. In the Security Center event logs, no events were posted indicating that any blocking had occurred. After searching for other issues with our client machines or content filters, we were able to solve the problem very simply by merely disabling AMP and re-enabling it shortly thereafter. The updates started flowing again just fine after resetting AMP in this way, and we have not had any issues with downloading legitimate .cab or .diagcab  files since. I'm not sure if there was some hang in the process that is supposed to be scanning .cab files or with the malware definitions in AMP, but toggling the enable configuration fixed the problem.

[Nash]

For this problem, the best solution is to install the 14.x beta firmware. It prevents AMP from getting "indigestion" and blocking downloads based on false positives.

My procedure is:

Restart AMP: disable it, save, wait for MX to update its config, then re-enable it.

Enable beta firmware under Network wide->General and schedule an update for maintenance window via Organization->Firmware Updates.