Upgrade firmware on a network with two switchs loses cloud connection and gets 1.1.1.1 address

CharlieCrackle
Building a reputation

Upgrade firmware on a network with two switchs loses cloud connection and gets 1.1.1.1 address

 

 

I upgraded a network with 2 x MS120 switches to  12.28 (stable)   and then lost the switches and connection to cloud.

 

Just before having to drive 80km to power cycle them  the MX started complaining about Duplicate IP 1.1.1.1  and the two mac addresses were the switches.

 

Why does the meraki use a public ip of 1.1.1.1   ?

 

Why do the switches boot with this ip ?  (I have them set for DHCP)

 

I did a wireshark capture and both switches are arping 8.8.8.8   and getting no where.....

 

can some one explain what is going on here    makes no sence..

 

 

wiresharkwireshark

 

 

19 REPLIES 19
PhilipDAth
Kind of a big deal
Kind of a big deal

I've never seen a switch default to using 1.1.1.1.  Are they statically configured?  Perhaps double check their IP configuration.

 

ps.  The switches should auto roll back the upgrade if they can't communicate with the Meraki cloud.  So try waiting a bit longer (30 minutes) before jumping into the car.

1.1.1.1 isn't Meraki thats Cloudflares DNS.

 

What are your MX DNS settings?

the MX is set to use Cisco umbrella DNS.   but the arp ing shows 1.1.1.1  (mac of switch)  trying to arp for the address of 8.8.8.8.

 

Had this happen to another single switch site now.   this site the switch is still working  but not connected to cloud  it was interesting as a power cycle did not fix the issue but the users were not effected and switch was still passing traffic

8-12 hours later the switch started talking to cloud its self with the last error being "Disabled Switch Bad DNS"

 

 

I am wondering if this is a MX DHCP issue  as all these issue started when the MXs were upgraded to 15.42  from 14  to fix the "Unable to download configuration error" which was happening more frequently.

Snap156.pngDoing a wire shark capture using the MAC address of the switch  you can see it making DHCP requests on native vlan and other vlan and getting responses  !

 

I just upgraded my office switches to 14.16  and got a duplicate IP 1.1.1.1  alert  when doing it.   do the switches boot with address of 1.1.1.1 ???   something wierd is going on here.  anyone else getting 1.1.1.1 duplicate ip alerts ??Snap.jpg

cmr
Kind of a big deal
Kind of a big deal

I upgraded a stack of three MS210s to 14.16 three weeks ago and didn't have any issues.  Their management interfaces are set using DHCP, are you using static IPs for management?

cmr
Kind of a big deal
Kind of a big deal

I just upgraded another site with MS120s, 210s, 220s and 225s from 14.12 to 14.16.  No issues like you experienced @CharlieCrackle 

CharlieCrackle
Building a reputation

No all set to DHCP

I have upgraded MS120from 14.12 to 14.16 without me any issues.

 

The switches are set to get their IP addresses form DHCP on a management VLAN.

 

Are you switches on a the default VLAN, or one you specified?

Dave Anderson
cmr
Kind of a big deal
Kind of a big deal

@DHAnderson I've upgraded switches with static and DHCP management interface IPs, some on default VLAN and some not (the ones that are not were all on DHCP).  But none were MS125s, I wonder if the issue applies to them?

DHAnderson
Head in the Cloud

@cmr ,

 

@CharlieCrackle is having issues with MS120, not MS125

 

I have lost a MS when the management VLAN set in the switch wasn't on the uplink port's allowed VLAN list. The switch was working, it just couldn't get to the dashboard.

 

I wonder if the isn't something like that.

 

 

Dave Anderson
cmr
Kind of a big deal
Kind of a big deal

@DHAnderson I think he has MS125s, look at the screenshot of them with the red arrows on.

CharlieCrackle
Building a reputation

Yes issue seen on 120s  and 125s  and switches on DHCP and default VLAN 1

@CharlieCrackle

 

The uplink port is set to allow VLAN 1 or all VLANS?

Dave Anderson

Just an update for you all.  it has taken along time for support to get back to me.

Switches boot with an IP address 1.1.1.1   it then changes to the static address or the DHCP address.

Support can not tell me why they use 1.1.1.1  (some one else IP address) or how the switch could ever use this address to get to the cloud.   If you are upgrading 3 switches at a time   then they will all reboot and at the same time there will be 3 devices with 1.1.1.1  causing the MX to error with duplicate IP.

 

If some one from engineering could explain what the rational is it would be great.

 

no both switches are set to DHCP.   it has been 9 hours now and they are still doing this.  what is wierd is all the cameras and AP that are connected to the switch are showing as not connected to cloud also but are working ok.  and the users on the switch are still working ok.  I don't get it at all and at this point support don't either.  Some one is going to site to power cycle the switches.

Gareth_
Conversationalist

Charlie did you ever get to the bottom of this? We have MS120's with this behaviour.

CharlieCrackle
Building a reputation

No I only know they boot with 1.1.1.1   address  no one has explained the reason.  I just upgrade 1 switch at a time now

Thanks Charlie. Just supporting your information adding that our switches have this behaviour also. Whilst we were resolving another issue regarding a DNS change in our network it is something that baffled us when it appeared in our logs.

Get notified when there are additional replies to this discussion.
Welcome to the Meraki Community!
To start contributing, simply sign in with your Cisco account. If you don't yet have a Cisco account, you can sign up.
Labels