Security Center

[JessIT1]

Morning all:  Saw this Source - IP with Action - Allowed on (2) of our firewall's external WAN IP's as the Destination.

Not sure why it was allowed..

I immediately added  95.214.52.173:500 & 95.214.52.173 to our Meraki Content Filtering - URL Filtering Block List and added a Layer 7 Deny rule for 95.214.52.173:500

[RaphaelL]

Hi , 

What are your threat protection settings ? : 

[JessIT1]

All of my MX Firewalls have this setting.

[JessIT1]

I believe these are the highest/most robust settings..

[RaphaelL]

 They are. Either the CVE score of vuln is so low that It doesn't get blocked by the IPS or there is a bug. I would suggest you to contact the Support.

[JessIT1]

Ok. Yes I did open a case this morning about it. thanks for your input

[JessIT1]

Still waiting to hear back from Meraki Support on this. Virus Total has 10 hits on the IP, so now sure how Action was allowed via IDS. Since attempts were at our MX firewall external WAN IP's not sure what damage could be done..

please advise. thanks

[acaudill]

Have you heard back yet? I had the same alerts this morning and am also waiting to hear back from Meraki Support. 

Thanks.

[JessIT1]

Ok, so the exact same IP address and Zyxel under details? No haven’t heard back yet.

[acaudill]

Yep the exact same IP and details.

[JessIT1]

From Meraki Support:

Thank you for contacting Cisco Meraki Technical Support! We understand that you are having issues with the source IP 95.214.52.173. However, would you be able to tell us when you saw this IP on your Firewall's external WAN IP's. In addition, by you adding it to your URL block list the firewall will block the traffic coming from outside to your MX. Therefore, if a device with that IP address is trying to access your MX it will be blocked. Now if there is a device in your network that is trying to access that IP they will be able to since, the MX blocks traffic coming in not going out.

[JessIT1]

So I sent support the times the IP was allowed, their response doesn't really address why it was allowed since it's a pretty high threat on Virus Total. I even added under Content Categories to Block; Regional Restricted Sites for all the ones available, Great Britain, Italy, Germany, Poland (IP 95.214.52.173 looked to come from Poland)

[JessIT1]

Thank you for contacting Cisco Meraki Technical Support.  We understand that you are obtaining traffic from that source IP and you would like this IP blocked from accessing your MX. In order to do this you will have to go to Security & SD-WAN and go to Security Center. Here you will be able to go to MX Events, and then here in this page you will be able to see the IDS alert that you received. If you were to click where it says "Zyxel unauthenticated IKEv2 overflow attempt", here you will be able to disable whitelist. After doing this you will no longer see this allowed. If you're still experiencing issues feel free to reach back to us.

My response:The whitelist on this Zyxel event is already set to Off, so of course I would not want to set to On to whitelist (allow). The issue is Meraki IDS allowed in the first place instead of blocking.

[ABR1988]

Hi,

We had the exact same alerts at the exact same time from the exact same IP Address. We've logged a ticket with Meraki but haven't had anything helpful back yet.

We dont have Zyxel devices but with the port being 500 (Ike) and it hitting a meraki mx public ip address on a meraki device that has the client vpn functionality enabled, we're definintely worried that the action says it was allowed. We've turned off the client vpn until we get an acceptable answer from Meraki. We also had IDS set to prevention and the mode as security before the alert happened.

Similar, we want to know why it was allowed. We've blocked it at layer 7 now but we need to know if the action was actually allowed as then we've been compromised which means that they may already have persistence in our network

Its suspicous that 3 users all with meraki devices have seen these exact alerts though. I'm going to update our ticket with this information to hopefully get someone on the case to connect the dots.

Thanks

[TechNick92]

Hi, we saw the same events on some appliances - not all we manage. Also created a layer 7 rule to block the ip.

We also have the same thoughts about this topic - why was the traffic allowed? Is there any kind of compromise?  And so on ...

Kind regards

[JessIT1]

As far as gaining access via our external public IP's, with a strong password and 2FA on our Meraki MX's don't see how they could have authenticated successfully. Also I checked logs for any rogue VPN connections from the IP in question at the time of the alert, not finding anything..

[ABR1988]

Do you have the client vpn enabled by any chance? I checked our VPN connections and radius server too and didn't see anything either but I'm unsure if that would show anything anyway if its an unauthenticated attack?

[JessIT1]

Yes we do have VPN enabled on 2 of 4 of our MX's. I have disabled for now as well, good idea on that.

[JessIT1]

Anyone else got any feedback from Meraki support, having VPN disabled is not ideal of course, remote users are barking.

[acaudill]

Nothing constructive from Meraki yet, but I did get some new alerts that were also allowed. The destination on these is also my firewall's external WAN IP. Not sure if this is related, but it seems pretty coincidental if not. Has anyone else seen these new alerts?

[JessIT1]

I've seen those before, but so far they have always shown blocked not allowed. Support definitely needs to escalate this issue as it's starting to affect more and more customers. Are they false positives or actual allowed attacks..

[GoMariners]

I haven't seen those specific alerts, but we are seeing other new ids alerts that started a couple of days ago. Our source and destination were all internal (esxi server to specific vm mostly). I called Meraki support and was told that snort updated and is now tagging incoming and outgoing traffic, rather than just incoming traffic. They said that they have received other calls about similar alerts and that after confirming that the traffic is legitimate, I can whitelist the alerts.

I'm not sure how accurate that is but we for sure are seeing new alerts for internal traffic.

[JessIT1]

Ok, but support still needs to give an explanation of alerts like these, did an unauthenticated user get access to our external WAN IP's. I have my VPN disabled for now as a precaution but remote users are barking about when I can re-enable.

[JessIT1]

I'm seeing alerts like this from an internal Windows 10 PC, no clue what Destination 255.255.255.255:5093 means..logs it ever hour

[cmr]

@JessIT1 255.255.255.255 is the broadcast address.  The source device is trying to find something in its subnet to respond.

Port 5093 is commonly used by RMS Sential License Manager, do you run that on your network, or does the machine sending the broadcast have a Sentinel client on it?

[JessIT1]

I'm throwing everything at the wall, added this layer 7 rule, IP source in my issue showed from Poland, but I added some other countries too..random I know but I figured why not.

[Vbrites]

My company has one Meraki MX and we are facing the same issue.
We have notice that event alerts in security center as well! And the notification is exactly the same! The same remote IP, similar hour and the same attack. Our meraki IDS also allowed the traffic.

However I have not oppened a ticket yet. I'll do this now.

As counter actions I have applied layer 7 rules blocking this remote IP, Region, created a layer3 rule to isolate the VPN subnet in order to log if any traffic will occour (while keeping the VPN client active), and I also added a layer 3 rule denying this remote IP in my ISP router (because meraki is not acctually facing the internet, it is NATed with port redirection.

If you guys have any ideas on that topic, I'll be glad to hear.

I'm keeping the VPN active in order to log in my syslog server any comunication that my occour. Until this moment, I checked every network, event logs and radius server logs and found no suspicious connection. I hope this is a false positive... Cant understand why IDS would allow an attack like this.

[JessIT1]

How did you create a layer3 rule to isolate the VPN subnet? Thanks 

[Vbrites]

I just added a rule to guarantee that no device connected to the VPN Client could comunicate with any other device, this include the internet itself:

The Idea is:

Deny - Any Protocol - From {VPN subnet} - At any src port - To any  Destination - At any dst port - syslog checked.

[JessIT1]

So even though you left the VPN enabled, adding that layer3 rule to deny any internet traffic on your VPN subnet, won’t users that connect to VPN not be able to get to the Internet then?

[Vbrites]

After set the layer3 rule I tested and noticed that the VPN user is still able to access the internet (what makes no sense to me) but the intervlan comunication was blocked. So what I saw is that the layer3 wont prevent the VPN user to access the internet, but it will prevent them to access your other subnets... So if you have servers and computers in other Vlan, the VPN user wont be able to access them.

I could not understand why internet acess was still happening even after adding the layer3 rule.

[JessIT1]

Ok, thanks for confirming that. One thing I did find out after adding the layer3 rule is that when I connect to our VPN over my phone to access my security cameras that was blocked until I removed the rule. Which does go along with what you were talking about as the cameras are on a separate internal subnet. So at least if a bad actor did connect to our VPN, they wouldn’t be able to attempt to access any of our internal computers or servers, which is a good thing. Thanks

[JessIT1]

So I just tested connecting to VPN, as soon as I connected my internet went to blocked and I could not Remote Desktop into my work computer, so back to square one, I will need to remove that layer3 deny for the VPN subnet if I want to leave VPN enabled.

[Vbrites]

I think this print will be more usefull:

[JessIT1]

From support:  We understand that you are expecting issues with 95.214.52.173 on port 500. However, there has been a recent update on the IDS/IPS ruleset and an improvement in terms of coverage and detection efficacy is expected. As result, certain traffic that before would not trigger an alert may now result in one.

that’s all fine and good but still doesn’t explain how an unauthenticated attempt from a foreign IP was allowed..

[ABR1988]

Ive had an almost identical response. Multiple users getting the same alert from the same ip at the same time is surely a cause for concern. I understand they've updated their snort ruleset so theres alot of noise but its a very specfic alert in my eyes. We have no links with the ip address so there shouldn't be any attempts on port 500 from them, especially allowed ones. I'm going to ask them to esculate it and see what comes of it

[ABR1988]

Hi All,

I've got a more thorough response from Meraki 

"Whatever was trying to connect to you over IKEv2 eventually stopped sending data. When this happens while the MX is detecting the event, it will flag it as malicious but will also flag it as "allowed" because nothing is generating data through that flow it inspected anymore. This can be disregarded as you aren't peering with a remote location that has that IP, so no tunnel would form, and client VPN doesn't use IKEv2 for VPN on top of you having it disabled."

It looks like the Meraki VPN uses L2TP while this attack was targetting IKEv2 which uses port 500 which L2TP doesn't use. That on top of the explanation around the allowed behaviour is comforting.

[Vbrites]

This may not be true, because documentation makes clear that is necessary that port 500 is opened.

https://documentation.meraki.com/MX/Client_VPN/Guided_Client_VPN_Troubleshooting#Resolving_NetBIOS_n...

[Vbrites]

After confirmation, this seems right... Client vpn uses 500 port but no IKEv2. Someone correct me if I'm wrong. 

[ABR1988]

Apologies, my poor research. It does use port 500 but not IKEv2.

However if you use the site to site vpn for sd wan connectivity it uses IKEv2 . Other site to site vpn to non meraki vpn peers can also use IKEv2. 

Hopefully meraki support give you the reassuring answer about the IDS allow action not being as it seems. I've asked for a bit more information about it too

[Vbrites]

I've got a support answer, but they did not give and explanation about the IDS allowing this attack:

"

Hi Victor,

Sorry to hear that you have been going through this. If you haven't already, please take the steps below for safety measures:

1. Create Layer 3 Firewall Deny rule on outbound list to 95.214.52.173 to ensure nothing from your LAN replies to that IP.
2. Create Layer 7 Firewall Deny rule to Deny all traffic to/from Poland as this IP comes from Poland.
3. Reboot your MX appliance immediately to kill all existing TCP flows if they are alive.

I hope this helps.

Thanks,
Peter
Meraki Support"

Ill answer asking so informations about the allow action that the IDS took. 

[JessIT1]

At this point I am just leaving VPN enabled, I tried layer3 outbound rule to deny any traffic within the VPN subnet as a precaution, but it's then pointless to enable VPN again since user can't access internet, shared drives, etc.

[JessIT1]

Latest from support on my case

Thank you for contacting Cisco Meraki Technical Support.

We understand that you are having concerns over your MX being compromised, but after looking over some backend logs you should not worry about your MX being compromised. We could see that the alert was identified as malicious traffic, but it was dropped afterwards because it was no longer present. This caused dashboard to show the event as allowed. However, you should not have any issues with a breach on your Client VPN. If you're still experiencing issues feel free to reach back to us.

[Mustafa_Taqi]

This is the Scenario that you are most likely experiencing. 

• Meraki MX appliance received packets from the source IP address 95.214.52.173.
• The packets were copied to the IDS process for further analysis.
• The IDS flagged the flow as potentially harmful, as it matches the pattern of a known attack vector.
• Before the IDS could take preemptive action to drop the flow, the Meraki MX's inbound firewall rules had already dropped it. In your case, if it was enabled, Client VPN could've been the process that dropped the flow as the destination port is port 500
• As a result of the firewall's prompt action, the IDS process could not apply its own measures, which is why the Meraki Dashboard indicated the action as "Allowed."
• It is important to note that despite this indication, the flow was effectively blocked by the MX.

Key Takeaways:

• The swift response by the firewall prevented any action from being required on the part of the IDS.
• An "Allowed" status on the Meraki Dashboard could sometimes mean that the threat was blocked by other security layers, not that the traffic was permitted through the network.

[JessIT1]

Ok, thank you for the detailed response on this issue.

[Vbrites]

Thank you, Mustafa! Now, this is much clearer.

I have a suggestion: It would be incredibly useful if Meraki could implement an update to the dashboard that allows it to specify which component effectively handled a threat and report it, instead of simply marking it as "allowed." This enhancement would help avoid confusion and provide more detailed insights into the security measures in place.

[JessIT1]

I agree. I was the one that started this case last Tuesday and have been engaged with the community here and Meraki support ever since trying to determine what if anything malicious was allowed to access our MX public WAN's. Jess

[Mustafa_Taqi]

I highly encourage you to use the "Give Your Feedback" feature on your Meraki dashboard to share your suggestion. User feedback is greatly valued and often carries more weight than that from the Support team.

Give your feedback (previously Make a Wish) - Cisco Meraki Documentation

[Matt11]

Thanks for this info @Mustafa_Taqi  This was happening to us as well. FWIW Mustafa has helped provide support over the phone in the past for an outage/issue and is incredibly knowledgeable. 

[Ramkumar1]

@Mustafa_Taqi I would like to bring a couple of points before accepting the scenario you explained above.

• How come an inbound firewall rule will block the traffic if this traffic overrides through the ClientVPN? If the client VPN drops a VPN packet, no user can connect to the VPN.
• Additionally, I believe if this packet is indeed malicious, it should be blocked at the IPS level, not at another layer.

[Mustafa_Taqi]

The Client VPN Process will drop the flow if the packets are not Legitimate IKE packets or if IKE Authentication fails. 

With respect to your second point, the behavior is constrained by the Architecture.  

[Ramkumar1]

@Mustafa_TaqiIf the client VPN process drops the flow, I would expect to see logs in the Event Logs. I don't believe it's dropped at the client VPN's end.

"The behavior is constrained by the Architecture" What do you mean by this? It's unclear now, where it's actually dropped. 

[JessIT1]

Happened again, these IP's were allowed on (2) of my MX's external WAN IP's - 98.194.65.91:500; 98.195.67.12:500; 98.194.65.92:500

Zyxel unauthenticated IKEv2 command injection attempt

Zyxel unauthenticated IKEv2 overflow attempt

[acaudill]

Yep I got them again also. Meraki Support has been useless.

 

[Vbrites]

Yeah, I got the same events... 

We can't just live  our lives with this doubts... It's hard to see this events appearing with "allow" tags and don't feel worried. 

[JessIT1]

I have added to my original Meraki case about this issue 2 weeks ago with this new info, have not heard back yet..

From Snort site about this allowed action we are seeing.

Link to CVE listed on Zyxel site - https://www.cve.org/CVERecord?id=CVE-2023-28771

Makes no sense, is a public Zyxel firewall trying to access our Meraki MX's public WAN's..?

[ABR1988]

I think Meraki explained to me before that the SNORT ids rules inspect the packet to see how it is structured and compares it against known vulnerabilties. So you can often get misleading alerts or false positives (e.g. an alert saying that theirs an apple ios vulnerabilty detected when it isn't an apple device)

[acaudill]

Reply I just got from Meraki.

"Greetings, IDS allowing known malicious signatures usually happens when the MX sees a single packet that matches the signature, but the flow terminates because the remote party reset the connection, so the MX can’t take action to “block” the flow. The security center still logs the event as "allowed" but no real malicious traffic was allowed for the flow since it was terminated. Please let me know if you have any questions."

[JessIT1]

Ok, if this is indeed the case, Meraki needs to put a headline alert on every client's dashboard overview page or send out a mass email or something. I do find it curious that some random person coming from an IP in Poland the first round and now in Texas the second round is hitting numerous customers with at the exact same time with the exact same Zyxel unauthenticated IKEv2 command injection attempt  &  Zyxel unauthenticated IKEv2 overflow attempt alert. (I'm sure a lot more that just don't get on this community and report).

[cmr]

If that were the case then surely the status could be expired, ceased or something else, anything else, other than allowed, which is the worst possible result to give!

[ABR1988]

That would be helpful, otherwise everytime we see an allowed action, a support ticket will need to be raised for them to confirm it hasn't actually been allowed

[JessIT1]

Same alerts in the early morning today, with same IP as yesterday.

98.194.65.91:500 allowed on one of my MX (2) public IP’s

yesterday I added denied this IP in these locations on MX

Content filtering- block URL

Layer 3 outbound rule deny

Layer 7 deny remote IP range & port 

[ABR1988]

They've appeared again for us too. No response on the meraki ticket in nearly 24 hours either.

[JessIT1]

ok, crud. Ya in a real bind now if even after deny rules are added to firewall that the action still comes up allowed...

[JimLay]

I'm seeing same issue with same IP and also IP  98.195.67.12, both allowed.  with settings in threat mgmt set to prevention.

[JessIT1]

Ok, the list of customers experiencing this keeps getting longer since I opened this issue over 2 weeks ago, Meraki support keeps giving this answer or something similar, doesn't give me confidence yet that an external IP is gaining access via external WAN IP's.

IDS allowing known malicious signatures usually happens when the MX sees a single packet that matches the signature, but the flow terminates because the remote party reset the connection, so the MX can’t take action to “block” the flow. The security center still logs the event as "allowed" but no real malicious traffic was allowed for the flow since it was terminated.

[JessIT1]

Latest Meraki support reply:

Thank you for contacting Cisco Meraki Technical Support!

This situation happens sometimes on older models like the MX64/65/84/100 which run threat detection out-of-band. Traffic is duplicated over to the threat detection engine, which then examines the traffic and makes a decision to allow/block it. This situation emerges when the traffic flow ends before we make the decision to block - for example, if it's only a packet or two long. We're still making the decision to block, but since the flow is gone by that time, it gets logged as allowed since nothing was actually blocked.

This behavior is outlined in our documentation: "Intrusion prevention on the MX used to block triggering malicious packets is designed to be best effort. Subsequent packets within the same malicious flow will be blocked.

Below I have provided a document that goes over this in more detail: 
https://documentation.meraki.com/MX/Content_Filtering_and_Threat_Protection/Threat_Protection#Intrus...

My response:  I think the only thing that is concerning is when do I need to be actually aware a successful breach has happened if allowed doesn’t really mean allowed, when should I note it as a false positive vs an actual breach that someone has made it inside our network to potentially set off a ransomware attack?

[JessIT1]

More allowed similar IP’s with Zyxel, I keep denying in 3 locations on firewall and they keep getting allowed, am I compromised am I not? Meraki support won’t give a confident yes or no. Support is becoming unreliable as of late, creating a dark cloud over my head for over 3 weeks now, argh!

[Vbrites]

JessIT1, do you have an ISP router or another router/firewall before your meraki? 

If yes, I think the best way to dealing with this is to configure this router to only allow incoming conections to this redirected port if coming from your collaborators public IP. I solved this issue doing this. However, if your collaborators don't have a public IP, this means that you will need them to daily keep you updated about their current IP to include it in the ISP router allow list.

This is the topology that I mean:

{INTERNET} -> [ISP ROUTER]* -> [MERAKI CLIENT VPN]

[ISP ROUTER]* : Only allow collaborators public IPs and then Redirect 500 and 4500 ports to Meraki.

But make sure that the ISP Router doesnt have any service itself facing the internet. Its important that this ISP router doesn't represent a vulnerability with services from itself facing the internet.

If someone disagree, please complement this discussion.