Brief loss of connectivity at multiple sites

[harmankardon]

Anyone else just experience a brief loss of internet connectivity across multiple sites? We just experienced some sort of connectivity, still investigating, but it seems that multiple sites lost primary WAN connectivity and have also lost their backup cellular connections (all sites are MX67C spread over a large geographical area with different ISPs).

I only received a handful of email notifications, dashboard and notifications do not seem to agree. Sorry for the lack of detail but just wanted to get this out there quickly to see if anyone else is in a similar boat.

[RaphaelL]

Nope. 1600 sites all working fine. 

You see loss inside and outside of the VPN ? Any logs that would suggest a brief loss of connectivity ?

[harmankardon]

1600 sites, that sounds like fun!

It looks like the MX firewalls at each site possibly rebooted at roughly the same time, but it's hard to tell because there is no true "reboot" event for Meraki event logs. I'm seeing these same events at all sites:

Jul 6 11:03:13 Status Ethernet port carrier change device: port1, carrier: true

Jul 6 11:03:13 Status Ethernet port carrier change device: port0, carrier: true

[RaphaelL]

oof ! I would suggest opening a case ! You are right , you can't see the uptime nor the reboot events.

[jrsilvius · Accepted solution]

Not sure if it is related, but we updated all our MX devices to 18.107.2 two weeks ago, and I fought with support for a week about the fact that we were dropping the VPN tunnels between all our sites and our data center every 2hrs for about 50 sec, so it was disconnecting all our users from their terminal sessions. we ended up rolling back to 17.x on our Data Center MX devices and the issue went away.

[harmankardon]

Well mystery solved. Turns out, it was an issue with 18.107.2 but not because of a bug in the firmware or anything, it was because the update to 18.107.2 happened in the middle of the day lol. No clue how or why it got scheduled for 11am on a weekday, will need to look into that.

[GreenMan]

The updates are scheduled for whatever day and time you have configured under Network-wide > General > Upgrade window.     Also check the Local time zone is correct, in the same menu.

If the upgrade was automatically scheduled by the Dashboard, your Admins will have been notified in advance - and could have rescheduled or cancelled, via Organization > Firmware upgrades > Scheduled changes tab.

[drgnslyr]

I almost did that when scheduling firmware updates once, clicked on the 10:00 rather than scrolling down to the 22:00.  Luckily I caught it quick enough to get it rescheduled without interruption.