[Service Notice] Unexpected MX reboots

[GiacomoS]

Good day Community,

We are currently investigating reports of unexpected reboots with the MX platforms. 

At present, we don't have much details, but if you are encountering this, please keep an eye on this thread for information. 

Thank you for your patience.

Giac

[jimmyt234]

We are seeing this across multiple customers this morning - snort related perhaps?

It would be great if you could get something up on the Meraki status page too.

[timshott]

Have also seen the snort rules update in your event log? We have, minutes before the reboot..

[ShaunB93]

Yes rebooted preceded by:

Intrusion Detection Intrusion detection rules update snort_rules_version: 20250414-2221

then again after reboot 

[Frede_der]

Same. 

on MX250 and MX85 but not MX68

// Correction, I have reboots on MX68 as well, just need to check more of them. 

[SamBrown]

Oh we've had 68s reboot. Initially I thought it was just the 68s until some others started.
It's definitely Snort update

[dsf6536]

we had mx 68 as well 

example this one 

[LEEMARCK]

MX67 and MX95 rebooting as well, every 10 minutes on the clock.

[SamBrown]

MXs repeatedly rebooting? Any of our affected MXs have only gone down once each. I note some have the snort update over an hour ago and haven't actually rebooted.

[Jordan1]

As far as I can tell right now, I'm only seeing one reboot, and some devices have the updated snort version ( snort_rules_version: 20250414-2221) and I believe they have not rebooted. 

This is one of those situations where having the device uptime in the dashboard would be REALLY helpful...

[rstory]

Which firmware version are you running Lee?

[Frede_der]

Disable Intrusion detection and prevention stops the reboots. It is not a great solution, but Firewall rebooting is worse. 

// So far, uptime of 20 min with Intrusion detection and prevention disabled. 

[Jordan1]

@Frede_der Where are you seeing uptime? 

[Frede_der]

I'm just looking at connectivity at the summary page

Security & SD-WAN - Appliance Status - summary - Connectivity graph. That shows when the last reboot was, as the MX disconnected from the dashboard. 

[Jordan1]

Understood. Strangely, I'm not seeing any data under the summary/connectivity graph. 

[Frede_der]

[Jordan1]

Weird... I'm seeing some devices with one drop, but other devices with no drop in the connectivity graph. 

[Jordan1]

Correction, I'm seeing data, but not seeing every device rebooting (or disconnecting from the dashboard).

[AbdullahPracha]

You can see it under the summary page for the MX.

[rh21]

Factory reset the MX, no good. The Power LED Stays just orange

[Mark_B1]

Disabling IDS/IPS as a workaround stabilizes the MX

[GiacomoS]

Hey folks,

We are working on the status page update as well, we should have something up shortly. 

Currently, workarounds are being tested, please bear with us, so we can offer better insight.

Many thanks for your patience. 

[Graham_W]

This is affecting us across multiple sites but why do you not have anything of your status page. Cisco / Meraki this is poor to say the least.

[SamerAl]

Hi All, 

Please check the meraki status page for any updates regarding this issue :

https://status.meraki.net/history

Thanks 

[GiacomoS]

Hey folks,

status.meraki.net has been updated and reflective of the incident. 

Giac

[Nicky-Z]

HA deployment also effect by this incident.

[rstory]

Seeing reboots for MX68W, MX100, and MX250s

[RNR-ITmeko]

Seeing reboots on MX67 and MX250s. 

Also see the intrusion rules update 3-4 minutes before the MX os crashes. As our MX's is in HA setups, this crash causes failovers

luckily this only happend once pr location so far. about 7 minutes after the MX that crashes comes online, the intrusion rules update is reapplied

[Prij]

All of my MX250 devices have already been rebooted six times.

[OTGBob]

We have two sites affected with Primary and Spare MX-450 Disable IDS/IPS seemed to fix one location but not the other

[Beebs]

Only seeing reboots on MX250 and MX450s thankfully.

MX85 and MX67 currently no issues.

[Nicky-Z]

after disable IPS/IDS  MX85 stop reboot.

but, MX105/MX250 still reboot, the issue still exist , not resolved.

[DANSTEE]

We are experiencing this issue too with a HA configuration of two Meraki MX105.

Just before each Network Outage we see the following Events. "Intrusion Detection Rules Update" and then "Events Dropped" followed by a Network Outage. It comes back up a few mins later and then repeats itself. It looks like a problem with Snort Rules Version 20250414-2221

Apr 16 12:51:19	<removed>		Intrusion Detection	Intrusion detection rules update	snort_rules_version: 20250414-2221
Apr 16 12:39:41	<removed>		Intrusion Detection	Intrusion detection rules update	snort_rules_version: 20250414-2221
Apr 16 12:28:04	<removed>		Intrusion Detection	Intrusion detection rules update	snort_rules_version: 20250414-2221
Apr 16 12:23:03	<removed>		Intrusion Detection	Intrusion detection rules update	snort_rules_version: 20250414-2221
Apr 16 12:11:25	<removed>		Intrusion Detection	Intrusion detection rules update	snort_rules_version: 20250414-2221
Apr 16 11:59:46	<removed>		Intrusion Detection	Intrusion detection rules update	snort_rules_version: 20250414-2221
Apr 16 11:48:09	<removed>		Intrusion Detection	Intrusion detection rules update	snort_rules_version: 20250414-2221
Apr 16 11:36:19	<removed>		Intrusion Detection	Intrusion detection rules update	snort_rules_version: 20250414-2221
Apr 16 11:30:17	<removed>		Intrusion Detection	Intrusion detection rules update	snort_rules_version: 20250414-2221

Apr 16 12:51:41	<removed>		Events dropped	Events dropped	200 events were not logged.
Apr 16 12:40:04	<removed>		Events dropped	Events dropped	211 events were not logged.
Apr 16 12:28:27	<removed>		Events dropped	Events dropped	209 events were not logged.
Apr 16 12:23:25	<removed>		Events dropped	Events dropped	209 events were not logged.
Apr 16 12:11:47	<removed>		Events dropped	Events dropped	218 events were not logged.
Apr 16 12:00:08	<removed>		Events dropped	Events dropped	216 events were not logged.
Apr 16 11:48:31	<removed>		Events dropped	Events dropped	218 events were not logged.
Apr 16 11:42:45	<removed>		Events dropped	Events dropped	1129 events were not logged.
Apr 16 11:36:42	<removed>		Events dropped	Events dropped	201 events were not logged.
Apr 16 09:22:46	<removed>		Events dropped	Events dropped	149 events were not logged.
Apr 16 09:10:16	<removed>		Events dropped	Events dropped	43 events were not logged.
Apr 15 11:19:56	<removed>		Events dropped	Events dropped	11 events were not logged.
Apr 15 10:38:56	<removed>		Events dropped	Events dropped	13 events were not logged.
Apr 15 08:53:16	<removed>		Events dropped	Events dropped	38 events were not logged.

We were running MX 19.1.7.2 (Beta) and we just downgraded to MX 18.211.5.2 (Stable Release) to see if that resolves the issue also. If this fails, we will disable Intrusion Detection as suggested.

I can confirm that downgrading the Firmware version does not resolve the issue. We are experiencing the exact same problem on both MX 19.1.7.2 (Beta) and o MX 18.211.5.2 (Stable Release).

We are also closely monitoring the Meraki Incident Report for this here: Meraki Status - MX Unexpectedly Rebooting

We have disabled both Advanced Malware Protection (AMP) and Intrusion Detection and Prevention (IDP) from Security & SD-WAN --> Threat Protection.

So far after disabling AMP and IDP, we have not had any reoccurrences of the issue.

[Adonis22]

Hello,
When these issues were happening to your MX in HA, would it take both of the MX down at the same time? Or was alternating between the two? (One going down/ comes up than the other goes down and back up?)

[dsf6536]

we notice all the mx impacted have 

Intrusion detection rules update

[Jordan1]

Looks like this was the faulty SNORT version: snort_rules_version: 20250414-2221, correct? 

[jburkart]

.

[jburkart]

I'm just posting because I don't see many people posting about MX105s. My HA pair has rebooted eight times now. We are running beta firmware. After the last reboot, I turned off IDS/IPS to test that working theory out, as I saw rules get updated, and then the device rebooted shortly after. 

Update -- with our HA pair the dashboard was showing 1 device update / reboot lasted about 1 min but the other would update during reboot and than reboot itself too -- alerts were going crazy but actual issues not 100% as no one was in the office yet.....

Can confirm disabling IDS has prevented the reboots as a workaround

[Martin7777]

After disabling AMP & IDS/IPS we have no reboots anymore - MX250, MX85, MX95. MX84 seems to be unaffected. MX100 rebooted only once.

[KentA]

possibly an issue with snort rules  snort_rules_version: 20250414-2221.  These are the rules I see being applied prior to reboots.

Disabling IPS did stop the reboots for us, but that is not a great tradeoff, security vs functionality

[RNR-ITmeko]

Update. on our MX85 setup the intrusion rule update was applied 13:13 to MX1, and 13:27 to MX2. None of them has rebooted. Seems like MX85 isn't effected by this crash

[Mark_B1]

Unfortunately we've got dozens of MX85 impacted, disabling IDS stabilizes them though

[Mloraditch]

It seems inconsistent. One client had their 105s and 250s rebooting but checked several other 250s with no issues. 

[DANSTEE]

We found those that were not impacted, had IDS disabled already. Have you checked that?

[CPA1]

We have MX85s that are having the issue as well.  I've disabled IDS/IPS on those devices and we have been stable for about 10 minutes.  

As previously mentioned, if that is the fix, not an ideal workaround.

[Nithinrajp]

We are also facing the rebooting issue with Meraki MX105

[Harry_solm]

Any further update on Service restoration ETA?

[RaphaelL]

Also experiencing this issue. 

Can someone confirm they have the issue with any version from 18.211 to 19.2

[rstory]

I confirm 18.211 on multiple MX hardware is rebooting but none for the last 30 minutes plus.

[Nicky-Z]

disable IPS/IDS or not?

[rstory]

Not

[Nicky-Z]

ok, thanks for confirmation.

my devices running in 19.1.7.2, reboot every 10 minutes before disable IPS/IDS.

[Frede_der]

I have the issue on MX68, MX85, MX250 and MX250 in HA, all running 18.211.5.2

All reboots stopped after I disabled Intrusion detection and prevention

[Beebs]

Yes MX 18.211.5.2

Strangely MX 18.211.2 MX450 didn't reboot just shows no connectivity at 11:29-11:30 UK Time.

[RaphaelL]

Thank you all for the confirmation

[Nug-Nug]

i have same issue MX68,MX85 hardware is rebooting ever 10 Minutes and device under Meraki cannot connect internet until now

[KarstenI]

Same here on MX68 and MX75 with 18.211.5.2. My MX64 (18.107.12) was unaffected.

[TyShawn]

MX75

19.1.7.2 taking reboots. 

[Nicky-Z]

we are waiting for resolution and root cause... Now, our all MX devices are disabling IPS/IDS to stop crash...

[FrankHWB]

Only saw 1 reboot of my mx95s (3 of them) early this morning.  fw 18.211.2

[KentA]

It appears we are only seeing the consistent reboots with our Hub devices, is anyone else seeing similar behaviour?

I only see a single reboot for many of our spokes

[Beebs]

Spokes with MX250 and MX450 constant reboots.

Hub with MX450 lost connectivity but didn't reboot.

All our MX67s and MX85s been fine luckily.

[KentA]

We have a MX450 hub that is experiencing the issue, appears to be hit and miss in some instances

[Sunilmm]

Any updates on the restoration?

[nicdc01]

With AMP disabled but IDS/IPS Detection enabled we noticed issues. FW-18.211.2
Devices with this disabled didn't experience the restarts.

After we made this changed we did not experience issues with our MX devices but not sure if this coincides with any hot-fixes they are currently pushing.

Can confirm from our side as well that After Event Intrusion Detection Rules Updated - snort_rules_version: 20250414-2221; there was a VRRP transition event and the MX rebooted.

HA networks seemed relatively stable but others not so much.

[Adonis22]

In your HA network, is the issue occurring but just one HA device at a time, which leads to a more stable experience?

[nicdc01]

Somewhat more stable at least in my environment compared to the networks that are not setup in HA.
However that is just my anecdotal observation.

Assuming on our end they just happened to occur at different times.

[ET_]

It must be the snort update. Also here in the event log. Snort update already tried 8 times today .... 8 restarts.. 

[AbdullahPracha]

Looks like disabling the Advanced Malware Protection & IPS is working.

Security & SD-WAN ---> Threat Protection.

[AnythingHosted]

This appears to be happening for our client as well. MX67, only enterprise licence, used for auto-vpn. Rebooting every 10-20 minutes.

[Beebs]

IMPORTANT

Even if disabling Intrusion detection and prevention is the fix surely your risking your network's security hence I'm not disabling the feature.

[WindyTech]

We also have a customer with MX85 that reboots every 13 minutes

[MelanieW]

do we have an update on when this will be resolved?

[Bertolo]

[DANSTEE]

I am astonished at the lack of official communication from Meraki over such a length of time

[Sacoleiro]

We need a solution, thinking about change ALL my network devices here. 

[d0rk]

We have sites with MX105 running in HA mode. Most have stopped rebooting now, however we have two sites where the "Spare" MX took over and the original "Primary" MX has not come back online. 

At both of these sites the VPN tunnel back to our VMX is also down.

[jphamwd]

We have a mix of MX84, MX85, and MX250s.  Only our MX250's (HA but only primary rebooting) are rebooting every 10 minutes.  Some of our MX84 and MX85s rebooted once.  Currently have IDS/IPS disabled on the 250s.  

[Salvo]

Hi all, i don't if it could work for you.
We solved the reboot problem by disabling ids and ips for 20 minutes and the reenabling it in prevention/balanced mode.
We have an mx95
No more reboots occurred and we have ids and ips back on

[RaphaelL]

I think a "fix" was just pushed.  

We see that they pushed a older SNORT rule. snort_rules_version: 20250409-1715

[PetterLarsson]

Observing the same, no restarts since the last snort-package was pushed so far (40min and counting, before that reboot every 10-15min always after package 2221 was downloaded)

[RaphaelL]

An update has been posted

A mitigation is being deployed to our systems and we are continuing to monitor the results. Your MX device may experience a further onetime reboot now during this remediation.

It fits with what we saw

[Kalpesh_Rane]

where do we check these snort update logs?

[jimmyt234]

Just under the regular Network-wide>Event Log page, you can event type include for: Intrusion detection rules update 

[Kalpesh_Rane]

Thanks

[Bertolo]

Estou com um outro site MX67 na minha rede com firmware MX18.211.5.2 que não esta sofrendo com essas reinicializações, já os que estão com o Firmware MX 18.211.2 estão com esse problema de reinicialização.

[d0rk]

How are Meraki intending to push updates to MX HA devices which went offline and never came back up? We have sites running on the "Spare" MX at the moment as we lost the "Primary" during the initial outage.

[GiacomoS]

Hey folks,

As per status page, a fix is being rolled out, you may experience another reboot in the process. 

@d0rk , I would recommend if possible to grab the Support Data Bundle, perform a manual reboot and worst case a factory reset. If it doesn't recover, please do give us a call so we can investigate with the SDB. 

Many thanks!

Giac

[d0rk]

Couldn't get to the LSP unfortunately, the MX wasn't supplying an IP over the management port. In any case, a reboot seemed to fix the issue. Thanks!

[Bertolo]

Meraki Status

[CharlieCrackle]

It has been really lucky that it is very late in AUSTRALIA and most customers are closed..  only me that could not work out why internet was going down changing to  different internet  downgrading firmware..

Why does the event log not show a crash and reboot event...  on the connection graph it just shows them as internet outages..   but the reboots I did showed as outage due to reboot.   having this would have saved a lot of time.    I assume meraki is going to do a full root cause analysis of what went wrong and how to stop this happening again on such a scale...   (one of the DISADVANTAGES of Cloud updated system !) 

[cmr]

My MX75 with IDP on security mode updates the rules at 10:56, didn't have any reboots and then updated back to the older version at 14:47.  Interesting how some MXs has no issues yet others crashed out.  Hopefully we might find out what configuration was triggering it?

[Bertolo]

Por aqui parece que estabilizou a conexão.
Meraki Status

[Bertolo]

[GiacomoS · Accepted solution]

Hey Folks,

As per the update on status.meraki.net , we are now seeing restoration of the service. 

If you are still experiencing issues, please do reach out to Meraki Support. 

Thank you so much for bearing with us during this disruption. 

Giac

[SamerAl]

Hi all ,

The issue was fully resolved on Apr 16, 17:16 UTC . If there's anything else you need, feel free to reach out to our support team.

We apologize for any inconvenience this may have caused.

Best Regards 

[CharlieCrackle]

This issue did point out how vulnerable we are to Meraki.   I would like to know the root cause and what went wrong in the meraki testing and validation process to let this sort of thing happen.  Also, what changes is Meraki making to their systems to avoid this in the future. 

[cmr]

Good point @CharlieCrackle, perhaps an update rollback button, or a scheduler similar to the firmware updates.

[jphamwd]

Sorry to unearth this thread, but I've been experiencing double the device utilization on my HA MX250s (18.211.5.1) since this event.  I disabled IDS/IPS (prevention/security) when the reboots started and then re-enabled after hours.  Can people that had issues that day take a look at their device utilization in the past 30 days (Organization > Summary Report > A single network > select appliance) and see if there is a marked increase since that day?  I called this into support and they saw I changed my client tracking to Unique Client ID around that same time and blamed that, but we have another with MX250s that is not using UCI and are seeing it there as well.  Sent screenshots of the last thirty days for both networks and waiting for a response but curious what you all are seeing TIA

[Beebs]

I'd suggest as a minimum upgrading to MX 18.211.5.2 being the current Stable which we've been running on all our MX's including MX250 for past 5 weeks. 

OR take a punt and use the Stable release candidate MX 19.17.7.2 as in the screenshot where I've underlined for your reference. 

Thankfully last week we only had the issues with MX450 and MX250s and our MX67, and MX85 were unaffected.

[JanJonsson]

Hey, 

Any idea on when the RCA will be available ?

Best regards,

Jan

[SamerAl]

Hi JanJonson,

It is avilable now , pelase reply to the case you logged with support and ask for it, if you do not have an existing case, please log a case refering to the date of this incident and ask for it.

Best Regards 

[jimmyt234]

Thanks for the heads up - we have just received the RCA from support after asking.

[JanJonsson]

Hey Jimmy, 

Would you be so kind to share it here ?

Thanks, 

Jan 

[jimmyt234]

No sorry, it is shared with a confidentiality notice - you should raise a ticket with Meraki support and they will send it with you.

[JanJonsson]

No problem, already done. Was just to speed up the process.

[JanJonsson]

Hey Giac, 

Can you kindly assist with a RCA on this ? I was mentioned it qwould be available within 5 business days.

Thanks, 

Best regards, 

Jan 

[GiacomoS]

Hey @JanJonsson , can you send me a direct message with the case number? I'll have a look 🙂 

Giac