MX67 Reboots Every Couple of Hours

[fxroche3]

Have an issue with MX67s rebooting like clock work every couple of hours, causing 10-12 minutes of downtime.

Site 1 - Multiple factory resets, upgrades/downgrades, even did a RMA and still had the same issue. Was told to bring it home and wire it up to my home network. Did that, brought it back to work and it no longer reboots. Not the ideal fix though.

Site 2 - Experiencing the same problem as Site 1, but don't have the luxury of bringing this one home since I'm 1000 miles away. Meraki wants to do another RMA, but its going to be a waste of time.

I have this plus 3 more remote sites I need to deploy, not to mention before Site 1 I did a deployment without a single problem. In all cases every site is using Palo Alto Firewalls and the rules for the MX are wide open and I validated that the Palo's aren't blocking or treating any of the traffic flows as Threats. In the case of Site 1 we even wired up the MX to our Internet Switch (no FW) and it still continued to reboot.

Anyone have any suggestions? Meraki Support is basically a dead end.

[NolanHerring]

1. Did you have support 100% confirm the MX67 is actually 'rebooting/power cycling'. You won't be able to tell that via the inadequate 'connectivity bar'. Hopefully one day they will give us the ability to see the 'uptime' column that support can see, until then you have to contact them for confirmation of the actual uptime of the device.

2. When you take it home, it works fine, and you have another site having the same issue. Me thinks this doesn't have anything to do with the MX67.

I would do a triple take on anything upstream from the MX appliance that would/could be 'blocking' cloud connectivity (assuming it isn't an actual power issue).

10 - 12 minutes of downtime is way longer than an actual power reboot takes, so I doubt its a power issue.

[fxroche3]

So the MX sits in its own DMZ and becomes unreachable for approx.10 mins every couple of hours. It's unreachable from inside the LAN and directly from the DMZ switch (interface goes down). On the FW there is also no traffic being logged so it definitely goes offline whether it's a reboot or not there is a link change on the switch interface. 

[PhilipDAth]

I haven't experienced this issue, but I normally have the Internet circuit plugged directly into the MX67.

Does the MX have its own public IP address, or does it have a private IP address and is the firewall in front of it NATing it to get to the Internet?

Is the MX67 running in NAT mode or VPN concentrator mode?

[fxroche3]

It has a private IP. Its 100% not a MX or network or firewall configuration issue. 

[PhilipDAth]

Is it in NAT or VPN concentrator mode?

When you had it at home did it go offline?

[PhilipDAth]

Are you running an up to date firmware like 14.39?

[fxroche3]

All versions to the current v13 to beta v14 and 15. Not a firmware problem, happens with all versions. Also it's worth mentioning that we didn't change one piece of any configuration after we brought it back from home. 

[fxroche3]

When this happened the first time Meraki Support said it looked like the firmware upgrade was not completing and therefore was attempting to redownload and install every couple of hours. Now I already dealt with Palo Alto and confirmed the firewall was not blocking any communications. So the only other difference between our corporate setup vs home setup is the Meraki is handed DHCP at home and using the cable providers DNS. Here we hard set the IP and use Google DNS, but again these are the same configurations in place today so why is it working now???

[fxroche3]

Nope and when we brought it back into the office it stayed connected and hasn't rebooted or gone offline since so something out its trying to do every couple of hours that for some reason is causing this issue. 

[fxroche3]

Just double checking my switch logs and there are link changes. Here is the most recent what I can only assume is a reboot. Dashboard shows offline from 6:42a-6:57a

Apr 2 06:44:36.429 est: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to down
Apr 2 06:44:37.428 est: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/23, changed state to down
Apr 2 06:44:40.389 est: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/23, changed state to up
Apr 2 06:44:42.388 est: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to up
Apr 2 06:45:26.984 est: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to down
Apr 2 06:45:27.984 est: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/23, changed state to down
Apr 2 06:45:31.067 est: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/23, changed state to up
Apr 2 06:45:33.066 est: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to up
Apr 2 06:55:57.825 est: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to down
Apr 2 06:55:58.828 est: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/23, changed state to down
Apr 2 06:56:01.869 est: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/23, changed state to up
Apr 2 06:56:03.868 est: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to up
Apr 2 06:56:37.269 est: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to down
Apr 2 06:56:38.268 est: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/23, changed state to down
Apr 2 06:56:41.187 est: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/23, changed state to up
Apr 2 06:56:43.190 est: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to up

[DavidLaz]

To this day still having this issue. Exactly as others have described it, same symptoms. Firmware rolled-back and re-updated, original MX RMA'd, tested it at another neighboring site and it works, brought it back to our office and left it online over the weekend, no reboots or WAN drops. The only thing we've been able to narrow it down to is the ISP. They had to change the client's public IP (DHCP) since the original wouldn't allow the Meraki to connect at all, it would just go through its rainbow cycle. Now it drops every 1.5 hours or so indicating there was an "unexpected reboot". Not bussin.