Security Center

JessIT1
Building a reputation

Security Center

Morning all:  Saw this Source - IP with Action - Allowed on (2) of our firewall's external WAN IP's as the Destination.

Not sure why it was allowed..

 

I immediately added  95.214.52.173:500 & 95.214.52.173 to our Meraki Content Filtering - URL Filtering Block List and added a Layer 7 Deny rule for 95.214.52.173:500

 

IDS1.jpgIDS2.jpg

IDS3.jpg

159 Replies 159
RaphaelL
Kind of a big deal
Kind of a big deal

Hi , 

 

What are your threat protection settings ? : 

 

RaphaelL_0-1707227170207.png

 

JessIT1
Building a reputation

All of my MX Firewalls have this setting.

IDS4.jpg

JessIT1
Building a reputation

I believe these are the highest/most robust settings..

RaphaelL
Kind of a big deal
Kind of a big deal

 They are. Either the CVE score of vuln is so low that It doesn't get blocked by the IPS or there is a bug. I would suggest you to contact the Support.

JessIT1
Building a reputation

Ok. Yes I did open a case this morning about it. thanks for your input

JessIT1
Building a reputation

Still waiting to hear back from Meraki Support on this. Virus Total has 10 hits on the IP, so now sure how Action was allowed via IDS. Since attempts were at our MX firewall external WAN IP's not sure what damage could be done..

 

please advise. thanks

 

IDS5.jpg

acaudill
Here to help

Have you heard back yet? I had the same alerts this morning and am also waiting to hear back from Meraki Support. 

 

Thanks.

JessIT1
Building a reputation

Ok, so the exact same IP address and Zyxel under details? No haven’t heard back yet.

JessIT1_0-1707260645121.jpeg

 

acaudill
Here to help

Yep the exact same IP and details.

JessIT1
Building a reputation

From Meraki Support:

 

Thank you for contacting Cisco Meraki Technical Support! We understand that you are having issues with the source IP 95.214.52.173. However, would you be able to tell us when you saw this IP on your Firewall's external WAN IP's. In addition, by you adding it to your URL block list the firewall will block the traffic coming from outside to your MX. Therefore, if a device with that IP address is trying to access your MX it will be blocked. Now if there is a device in your network that is trying to access that IP they will be able to since, the MX blocks traffic coming in not going out.

JessIT1
Building a reputation

So I sent support the times the IP was allowed, their response doesn't really address why it was allowed since it's a pretty high threat on Virus Total. I even added under Content Categories to Block; Regional Restricted Sites for all the ones available, Great Britain, Italy, Germany, Poland (IP 95.214.52.173 looked to come from Poland)

JessIT1
Building a reputation

Thank you for contacting Cisco Meraki Technical Support.  We understand that you are obtaining traffic from that source IP and you would like this IP blocked from accessing your MX. In order to do this you will have to go to Security & SD-WAN and go to Security Center. Here you will be able to go to MX Events, and then here in this page you will be able to see the IDS alert that you received. If you were to click where it says "Zyxel unauthenticated IKEv2 overflow attempt", here you will be able to disable whitelist. After doing this you will no longer see this allowed. If you're still experiencing issues feel free to reach back to us.

 

My response:The whitelist on this Zyxel event is already set to Off, so of course I would not want to set to On to whitelist (allow). The issue is Meraki IDS allowed in the first place instead of blocking.

ABR1988
Here to help

Hi,

 

We had the exact same alerts at the exact same time from the exact same IP Address. We've logged a ticket with Meraki but haven't had anything helpful back yet.

 

We dont have Zyxel devices but with the port being 500 (Ike) and it hitting a meraki mx public ip address on a meraki device that has the client vpn functionality enabled, we're definintely worried that the action says it was allowed. We've turned off the client vpn until we get an acceptable answer from Meraki. We also had IDS set to prevention and the mode as security before the alert happened.

 

Similar, we want to know why it was allowed. We've blocked it at layer 7 now but we need to know if the action was actually allowed as then we've been compromised which means that they may already have persistence in our network

 

Its suspicous that 3 users all with meraki devices have seen these exact alerts though. I'm going to update our ticket with this information to hopefully get someone on the case to connect the dots.

 

Thanks

TechNick92
Here to help

Hi, we saw the same events on some appliances - not all we manage. Also created a layer 7 rule to block the ip.

We also have the same thoughts about this topic - why was the traffic allowed? Is there any kind of compromise?  And so on ...

Kind regards

 

JessIT1
Building a reputation

As far as gaining access via our external public IP's, with a strong password and 2FA on our Meraki MX's don't see how they could have authenticated successfully. Also I checked logs for any rogue VPN connections from the IP in question at the time of the alert, not finding anything..

ABR1988
Here to help

Do you have the client vpn enabled by any chance? I checked our VPN connections and radius server too and didn't see anything either but I'm unsure if that would show anything anyway if its an unauthenticated attack?

JessIT1
Building a reputation

Yes we do have VPN enabled on 2 of 4 of our MX's. I have disabled for now as well, good idea on that.

JessIT1
Building a reputation

Anyone else got any feedback from Meraki support, having VPN disabled is not ideal of course, remote users are barking.

acaudill
Here to help

Nothing constructive from Meraki yet, but I did get some new alerts that were also allowed. The destination on these is also my firewall's external WAN IP. Not sure if this is related, but it seems pretty coincidental if not. Has anyone else seen these new alerts?

 

Screenshot 2024-02-08 at 10.35.55 AM.pngScreenshot 2024-02-08 at 10.36.22 AM.png

JessIT1
Building a reputation

I've seen those before, but so far they have always shown blocked not allowed. Support definitely needs to escalate this issue as it's starting to affect more and more customers. Are they false positives or actual allowed attacks..

GoMariners
New here

I haven't seen those specific alerts, but we are seeing other new ids alerts that started a couple of days ago. Our source and destination were all internal (esxi server to specific vm mostly). I called Meraki support and was told that snort updated and is now tagging incoming and outgoing traffic, rather than just incoming traffic. They said that they have received other calls about similar alerts and that after confirming that the traffic is legitimate, I can whitelist the alerts.

 

I'm not sure how accurate that is but we for sure are seeing new alerts for internal traffic.

 

Screenshot 2024-02-08 110846.png

JessIT1
Building a reputation

Ok, but support still needs to give an explanation of alerts like these, did an unauthenticated user get access to our external WAN IP's. I have my VPN disabled for now as a precaution but remote users are barking about when I can re-enable.


IDS6.jpgIDS2.jpg

JessIT1
Building a reputation

I'm seeing alerts like this from an internal Windows 10 PC, no clue what Destination 255.255.255.255:5093 means..logs it ever hour

 

IDS8.jpg

cmr
Kind of a big deal
Kind of a big deal

@JessIT1 255.255.255.255 is the broadcast address.  The source device is trying to find something in its subnet to respond.

 

Port 5093 is commonly used by RMS Sential License Manager, do you run that on your network, or does the machine sending the broadcast have a Sentinel client on it?

If my answer solves your problem please click Accept as Solution so others can benefit from it.
JessIT1
Building a reputation

IDS7.jpg

I'm throwing everything at the wall, added this layer 7 rule, IP source in my issue showed from Poland, but I added some other countries too..random I know but I figured why not.

Vbrites
Getting noticed

My company has one Meraki MX and we are facing the same issue.
We have notice that event alerts in security center as well! And the notification is exactly the same! The same remote IP, similar hour and the same attack. Our meraki IDS also allowed the traffic.

 

Vbrites_0-1707436956463.pngVbrites_1-1707436969004.png

 

However I have not oppened a ticket yet. I'll do this now.

As counter actions I have applied layer 7 rules blocking this remote IP, Region, created a layer3 rule to isolate the VPN subnet in order to log if any traffic will occour (while keeping the VPN client active), and I also added a layer 3 rule denying this remote IP in my ISP router (because meraki is not acctually facing the internet, it is NATed with port redirection.

If you guys have any ideas on that topic, I'll be glad to hear.

I'm keeping the VPN active in order to log in my syslog server any comunication that my occour. Until this moment, I checked every network, event logs and radius server logs and found no suspicious connection. I hope this is a false positive... Cant understand why IDS would allow an attack like this.

JessIT1
Building a reputation

How did you create a layer3 rule to isolate the VPN subnet? Thanks 

Vbrites
Getting noticed

I just added a rule to guarantee that no device connected to the VPN Client could comunicate with any other device, this include the internet itself:

Vbrites_0-1707438310142.png

The Idea is:

Deny - Any Protocol - From {VPN subnet} - At any src port - To any  Destination - At any dst port - syslog checked.

JessIT1
Building a reputation

So even though you left the VPN enabled, adding that layer3 rule to deny any internet traffic on your VPN subnet, won’t users that connect to VPN not be able to get to the Internet then?

Vbrites
Getting noticed

After set the layer3 rule I tested and noticed that the VPN user is still able to access the internet (what makes no sense to me) but the intervlan comunication was blocked. So what I saw is that the layer3 wont prevent the VPN user to access the internet, but it will prevent them to access your other subnets... So if you have servers and computers in other Vlan, the VPN user wont be able to access them.

 

I could not understand why internet acess was still happening even after adding the layer3 rule.

JessIT1
Building a reputation

Ok, thanks for confirming that. One thing I did find out after adding the layer3 rule is that when I connect to our VPN over my phone to access my security cameras that was blocked until I removed the rule. Which does go along with what you were talking about as the cameras are on a separate internal subnet. So at least if a bad actor did connect to our VPN, they wouldn’t be able to attempt to access any of our internal computers or servers, which is a good thing. Thanks

JessIT1
Building a reputation

So I just tested connecting to VPN, as soon as I connected my internet went to blocked and I could not Remote Desktop into my work computer, so back to square one, I will need to remove that layer3 deny for the VPN subnet if I want to leave VPN enabled.

Vbrites
Getting noticed

I think this print will be more usefull:

Vbrites_1-1707438479714.png

 

JessIT1
Building a reputation

From support:  We understand that you are expecting issues with 95.214.52.173 on port 500. However, there has been a recent update on the IDS/IPS ruleset and an improvement in terms of coverage and detection efficacy is expected. As result, certain traffic that before would not trigger an alert may now result in one.

 

that’s all fine and good but still doesn’t explain how an unauthenticated attempt from a foreign IP was allowed..

ABR1988
Here to help

Ive had an almost identical response. Multiple users getting the same alert from the same ip at the same time is surely a cause for concern. I understand they've updated their snort ruleset so theres alot of noise but its a very specfic alert in my eyes. We have no links with the ip address so there shouldn't be any attempts on port 500 from them, especially allowed ones. I'm going to ask them to esculate it and see what comes of it

ABR1988
Here to help

Hi All,

 

I've got a more thorough response from Meraki 


"Whatever was trying to connect to you over IKEv2 eventually stopped sending data. When this happens while the MX is detecting the event, it will flag it as malicious but will also flag it as "allowed" because nothing is generating data through that flow it inspected anymore. This can be disregarded as you aren't peering with a remote location that has that IP, so no tunnel would form, and client VPN doesn't use IKEv2 for VPN on top of you having it disabled."

 

It looks like the Meraki VPN uses L2TP while this attack was targetting IKEv2 which uses port 500 which L2TP doesn't use. That on top of the explanation around the allowed behaviour is comforting.

Vbrites
Getting noticed

This may not be true, because documentation makes clear that is necessary that port 500 is opened.

 

https://documentation.meraki.com/MX/Client_VPN/Guided_Client_VPN_Troubleshooting#Resolving_NetBIOS_n...

Vbrites
Getting noticed

After confirmation, this seems right... Client vpn uses 500 port but no IKEv2. Someone correct me if I'm wrong. 

ABR1988
Here to help

Apologies, my poor research. It does use port 500 but not IKEv2.

 

However if you use the site to site vpn for sd wan connectivity it uses IKEv2 . Other site to site vpn to non meraki vpn peers can also use IKEv2. 

 

Hopefully meraki support give you the reassuring answer about the IDS allow action not being as it seems. I've asked for a bit more information about it too

Vbrites
Getting noticed

I've got a support answer, but they did not give and explanation about the IDS allowing this attack:

 

"

Hi Victor,

Sorry to hear that you have been going through this. If you haven't already, please take the steps below for safety measures:

1. Create Layer 3 Firewall Deny rule on outbound list to 95.214.52.173 to ensure nothing from your LAN replies to that IP.
2. Create Layer 7 Firewall Deny rule to Deny all traffic to/from Poland as this IP comes from Poland.
3. Reboot your MX appliance immediately to kill all existing TCP flows if they are alive.

I hope this helps.

Thanks,
Peter
Meraki Support"

 

Ill answer asking so informations about the allow action that the IDS took. 

JessIT1
Building a reputation

At this point I am just leaving VPN enabled, I tried layer3 outbound rule to deny any traffic within the VPN subnet as a precaution, but it's then pointless to enable VPN again since user can't access internet, shared drives, etc.

JessIT1
Building a reputation

Latest from support on my case

 

Thank you for contacting Cisco Meraki Technical Support.

We understand that you are having concerns over your MX being compromised, but after looking over some backend logs you should not worry about your MX being compromised. We could see that the alert was identified as malicious traffic, but it was dropped afterwards because it was no longer present. This caused dashboard to show the event as allowed. However, you should not have any issues with a breach on your Client VPN. If you're still experiencing issues feel free to reach back to us.

Mustafa_Taqi
Meraki Employee
Meraki Employee

This is the Scenario that you are most likely experiencing. 

 

  • Meraki MX appliance received packets from the source IP address 95.214.52.173.
  • The packets were copied to the IDS process for further analysis.
  • The IDS flagged the flow as potentially harmful, as it matches the pattern of a known attack vector.
  • Before the IDS could take preemptive action to drop the flow, the Meraki MX's inbound firewall rules had already dropped it. In your case, if it was enabled, Client VPN could've been the process that dropped the flow as the destination port is port 500
  • As a result of the firewall's prompt action, the IDS process could not apply its own measures, which is why the Meraki Dashboard indicated the action as "Allowed."
  • It is important to note that despite this indication, the flow was effectively blocked by the MX.

Key Takeaways:

  • The swift response by the firewall prevented any action from being required on the part of the IDS.
  • An "Allowed" status on the Meraki Dashboard could sometimes mean that the threat was blocked by other security layers, not that the traffic was permitted through the network.

 

JessIT1
Building a reputation

Ok, thank you for the detailed response on this issue.

Vbrites
Getting noticed

Thank you, Mustafa! Now, this is much clearer.

I have a suggestion: It would be incredibly useful if Meraki could implement an update to the dashboard that allows it to specify which component effectively handled a threat and report it, instead of simply marking it as "allowed." This enhancement would help avoid confusion and provide more detailed insights into the security measures in place.

JessIT1
Building a reputation

I agree. I was the one that started this case last Tuesday and have been engaged with the community here and Meraki support ever since trying to determine what if anything malicious was allowed to access our MX public WAN's. Jess

Mustafa_Taqi
Meraki Employee
Meraki Employee

I highly encourage you to use the "Give Your Feedback" feature on your Meraki dashboard to share your suggestion. User feedback is greatly valued and often carries more weight than that from the Support team.

 

Give your feedback (previously Make a Wish) - Cisco Meraki Documentation

 

Matt11
Conversationalist

Thanks for this info @Mustafa_Taqi  This was happening to us as well. FWIW Mustafa has helped provide support over the phone in the past for an outage/issue and is incredibly knowledgeable. 

Ramkumar1
New here

@Mustafa_Taqi I would like to bring a couple of points before accepting the scenario you explained above.

 

  • How come an inbound firewall rule will block the traffic if this traffic overrides through the ClientVPN? If the client VPN drops a VPN packet, no user can connect to the VPN.
  • Additionally, I believe if this packet is indeed malicious, it should be blocked at the IPS level, not at another layer.
Mustafa_Taqi
Meraki Employee
Meraki Employee

The Client VPN Process will drop the flow if the packets are not Legitimate IKE packets or if IKE Authentication fails. 

 

With respect to your second point, the behavior is constrained by the Architecture.  

Ramkumar1
New here

@Mustafa_TaqiIf the client VPN process drops the flow, I would expect to see logs in the Event Logs. I don't believe it's dropped at the client VPN's end.

"The behavior is constrained by the Architecture" What do you mean by this? It's unclear now, where it's actually dropped. 

JessIT1
Building a reputation

Happened again, these IP's were allowed on (2) of my MX's external WAN IP's - 98.194.65.91:500; 98.195.67.12:500; 98.194.65.92:500

 

Zyxel unauthenticated IKEv2 command injection attempt

 

Zyxel unauthenticated IKEv2 overflow attempt

 

acaudill
Here to help

Yep I got them again also. Meraki Support has been useless.

Screenshot 2024-02-21 at 2.54.06 PM.png 

Vbrites
Getting noticed

Yeah, I got the same events... 

 

We can't just live  our lives with this doubts... It's hard to see this events appearing with "allow" tags and don't feel worried. 

JessIT1
Building a reputation

I have added to my original Meraki case about this issue 2 weeks ago with this new info, have not heard back yet..

 

From Snort site about this allowed action we are seeing.

 

Link to CVE listed on Zyxel site - https://www.cve.org/CVERecord?id=CVE-2023-28771

 

Makes no sense, is a public Zyxel firewall trying to access our Meraki MX's public WAN's..?

 

IDS9.jpg

ABR1988
Here to help

I think Meraki explained to me before that the SNORT ids rules inspect the packet to see how it is structured and compares it against known vulnerabilties. So you can often get misleading alerts or false positives (e.g. an alert saying that theirs an apple ios vulnerabilty detected when it isn't an apple device)

acaudill
Here to help

Reply I just got from Meraki.

 

"Greetings, IDS allowing known malicious signatures usually happens when the MX sees a single packet that matches the signature, but the flow terminates because the remote party reset the connection, so the MX can’t take action to “block” the flow. The security center still logs the event as "allowed" but no real malicious traffic was allowed for the flow since it was terminated. Please let me know if you have any questions."

JessIT1
Building a reputation

Ok, if this is indeed the case, Meraki needs to put a headline alert on every client's dashboard overview page or send out a mass email or something. I do find it curious that some random person coming from an IP in Poland the first round and now in Texas the second round is hitting numerous customers with at the exact same time with the exact same Zyxel unauthenticated IKEv2 command injection attempt  &  Zyxel unauthenticated IKEv2 overflow attempt alert. (I'm sure a lot more that just don't get on this community and report).

cmr
Kind of a big deal
Kind of a big deal

If that were the case then surely the status could be expired, ceased or something else, anything else, other than allowed, which is the worst possible result to give!

If my answer solves your problem please click Accept as Solution so others can benefit from it.
ABR1988
Here to help

That would be helpful, otherwise everytime we see an allowed action, a support ticket will need to be raised for them to confirm it hasn't actually been allowed

JessIT1
Building a reputation

Same alerts in the early morning today, with same IP as yesterday.

 

98.194.65.91:500 allowed on one of my MX (2) public IP’s
 
yesterday I added denied this IP in these locations on MX
 
Content filtering- block URL
 
Layer 3 outbound rule deny
 
Layer 7 deny remote IP range & port 
ABR1988
Here to help

They've appeared again for us too. No response on the meraki ticket in nearly 24 hours either.

JessIT1
Building a reputation

ok, crud. Ya in a real bind now if even after deny rules are added to firewall that the action still comes up allowed...

JimLay
Comes here often

I'm seeing same issue with same IP and also IP  98.195.67.12, both allowed.  with settings in threat mgmt set to prevention.

JessIT1
Building a reputation

Ok, the list of customers experiencing this keeps getting longer since I opened this issue over 2 weeks ago, Meraki support keeps giving this answer or something similar, doesn't give me confidence yet that an external IP is gaining access via external WAN IP's.

 

IDS allowing known malicious signatures usually happens when the MX sees a single packet that matches the signature, but the flow terminates because the remote party reset the connection, so the MX can’t take action to “block” the flow. The security center still logs the event as "allowed" but no real malicious traffic was allowed for the flow since it was terminated.

 

 

JessIT1
Building a reputation

Latest Meraki support reply:

 

Thank you for contacting Cisco Meraki Technical Support!

This situation happens sometimes on older models like the MX64/65/84/100 which run threat detection out-of-band. Traffic is duplicated over to the threat detection engine, which then examines the traffic and makes a decision to allow/block it. This situation emerges when the traffic flow ends before we make the decision to block - for example, if it's only a packet or two long. We're still making the decision to block, but since the flow is gone by that time, it gets logged as allowed since nothing was actually blocked.

This behavior is outlined in our documentation: "Intrusion prevention on the MX used to block triggering malicious packets is designed to be best effort. Subsequent packets within the same malicious flow will be blocked.

 

Below I have provided a document that goes over this in more detail: 
https://documentation.meraki.com/MX/Content_Filtering_and_Threat_Protection/Threat_Protection#Intrus...

 

 

My response:  I think the only thing that is concerning is when do I need to be actually aware a successful breach has happened if allowed doesn’t really mean allowed, when should I note it as a false positive vs an actual breach that someone has made it inside our network to potentially set off a ransomware attack?

JessIT1
Building a reputation

More allowed similar IP’s with Zyxel, I keep denying in 3 locations on firewall and they keep getting allowed, am I compromised am I not? Meraki support won’t give a confident yes or no. Support is becoming unreliable as of late, creating a dark cloud over my head for over 3 weeks now, argh!

Vbrites
Getting noticed

JessIT1, do you have an ISP router or another router/firewall before your meraki? 

If yes, I think the best way to dealing with this is to configure this router to only allow incoming conections to this redirected port if coming from your collaborators public IP. I solved this issue doing this. However, if your collaborators don't have a public IP, this means that you will need them to daily keep you updated about their current IP to include it in the ISP router allow list.

This is the topology that I mean:

{INTERNET} -> [ISP ROUTER]* -> [MERAKI CLIENT VPN]

 

[ISP ROUTER]* : Only allow collaborators public IPs and then Redirect 500 and 4500 ports to Meraki.

But make sure that the ISP Router doesnt have any service itself facing the internet. Its important that this ISP router doesn't represent a vulnerability with services from itself facing the internet.

If someone disagree, please complement this discussion.