Brief loss of connectivity at multiple sites

Solved
harmankardon
Building a reputation

Brief loss of connectivity at multiple sites

Anyone else just experience a brief loss of internet connectivity across multiple sites? We just experienced some sort of connectivity, still investigating, but it seems that multiple sites lost primary WAN connectivity and have also lost their backup cellular connections (all sites are MX67C spread over a large geographical area with different ISPs).

 

I only received a handful of email notifications, dashboard and notifications do not seem to agree. Sorry for the lack of detail but just wanted to get this out there quickly to see if anyone else is in a similar boat.

1 Accepted Solution
jrsilvius
Getting noticed

Not sure if it is related, but we updated all our MX devices to 18.107.2 two weeks ago, and I fought with support for a week about the fact that we were dropping the VPN tunnels between all our sites and our data center every 2hrs for about 50 sec, so it was disconnecting all our users from their terminal sessions. we ended up rolling back to 17.x on our Data Center MX devices and the issue went away.

View solution in original post

7 Replies 7
RaphaelL
Kind of a big deal
Kind of a big deal

Nope. 1600 sites all working fine. 

 

You see loss inside and outside of the VPN ? Any logs that would suggest a brief loss of connectivity ?

harmankardon
Building a reputation

1600 sites, that sounds like fun!

 

It looks like the MX firewalls at each site possibly rebooted at roughly the same time, but it's hard to tell because there is no true "reboot" event for Meraki event logs. I'm seeing these same events at all sites:

 

Jul 6 11:03:13 Status Ethernet port carrier change device: port1, carrier: true
Jul 6 11:03:13 Status Ethernet port carrier change device: port0, carrier: true
RaphaelL
Kind of a big deal
Kind of a big deal

oof ! I would suggest opening a case ! You are right , you can't see the uptime nor the reboot events.

jrsilvius
Getting noticed

Not sure if it is related, but we updated all our MX devices to 18.107.2 two weeks ago, and I fought with support for a week about the fact that we were dropping the VPN tunnels between all our sites and our data center every 2hrs for about 50 sec, so it was disconnecting all our users from their terminal sessions. we ended up rolling back to 17.x on our Data Center MX devices and the issue went away.

harmankardon
Building a reputation

Well mystery solved. Turns out, it was an issue with 18.107.2 but not because of a bug in the firmware or anything, it was because the update to 18.107.2 happened in the middle of the day lol. No clue how or why it got scheduled for 11am on a weekday, will need to look into that.

GreenMan
Meraki Employee
Meraki Employee

The updates are scheduled for whatever day and time you have configured under Network-wide > General > Upgrade window.     Also check the Local time zone is correct, in the same menu.

If the upgrade was automatically scheduled by the Dashboard, your Admins will have been notified in advance - and could have rescheduled or cancelled, via Organization > Firmware upgrades > Scheduled changes tab.

drgnslyr
Getting noticed

I almost did that when scheduling firmware updates once, clicked on the 10:00 rather than scrolling down to the 22:00.  Luckily I caught it quick enough to get it rescheduled without interruption.

Get notified when there are additional replies to this discussion.