Upgrade firmware on a network with two switchs loses cloud connection and gets 1.1.1.1 address

[CharlieCrackle]

I upgraded a network with 2 x MS120 switches to  12.28 (stable)   and then lost the switches and connection to cloud.

Just before having to drive 80km to power cycle them  the MX started complaining about Duplicate IP 1.1.1.1  and the two mac addresses were the switches.

Why does the meraki use a public ip of 1.1.1.1   ?

Why do the switches boot with this ip ?  (I have them set for DHCP)

I did a wireshark capture and both switches are arping 8.8.8.8   and getting no where.....

can some one explain what is going on here    makes no sence..

wireshark

[PhilipDAth]

I've never seen a switch default to using 1.1.1.1.  Are they statically configured?  Perhaps double check their IP configuration.

ps.  The switches should auto roll back the upgrade if they can't communicate with the Meraki cloud.  So try waiting a bit longer (30 minutes) before jumping into the car.

[BlakeRichardson]

1.1.1.1 isn't Meraki thats Cloudflares DNS.

What are your MX DNS settings?

[CharlieCrackle]

the MX is set to use Cisco umbrella DNS.   but the arp ing shows 1.1.1.1  (mac of switch)  trying to arp for the address of 8.8.8.8.

Had this happen to another single switch site now.   this site the switch is still working  but not connected to cloud  it was interesting as a power cycle did not fix the issue but the users were not effected and switch was still passing traffic

8-12 hours later the switch started talking to cloud its self with the last error being "Disabled Switch Bad DNS"

I am wondering if this is a MX DHCP issue  as all these issue started when the MXs were upgraded to 15.42  from 14  to fix the "Unable to download configuration error" which was happening more frequently.

[CharlieCrackle]

Doing a wire shark capture using the MAC address of the switch  you can see it making DHCP requests on native vlan and other vlan and getting responses  !

[CharlieCrackle]

I just upgraded my office switches to 14.16  and got a duplicate IP 1.1.1.1  alert  when doing it.   do the switches boot with address of 1.1.1.1 ???   something wierd is going on here.  anyone else getting 1.1.1.1 duplicate ip alerts ??

[cmr]

I upgraded a stack of three MS210s to 14.16 three weeks ago and didn't have any issues.  Their management interfaces are set using DHCP, are you using static IPs for management?

[cmr]

I just upgraded another site with MS120s, 210s, 220s and 225s from 14.12 to 14.16.  No issues like you experienced @CharlieCrackle 

[CharlieCrackle]

No all set to DHCP

[DHAnderson]

I have upgraded MS120from 14.12 to 14.16 without me any issues.

The switches are set to get their IP addresses form DHCP on a management VLAN.

Are you switches on a the default VLAN, or one you specified?

[cmr]

@DHAnderson I've upgraded switches with static and DHCP management interface IPs, some on default VLAN and some not (the ones that are not were all on DHCP).  But none were MS125s, I wonder if the issue applies to them?

[DHAnderson]

@cmr ,

@CharlieCrackle is having issues with MS120, not MS125

I have lost a MS when the management VLAN set in the switch wasn't on the uplink port's allowed VLAN list. The switch was working, it just couldn't get to the dashboard.

I wonder if the isn't something like that.

[cmr]

@DHAnderson I think he has MS125s, look at the screenshot of them with the red arrows on.

[CharlieCrackle]

Yes issue seen on 120s  and 125s  and switches on DHCP and default VLAN 1

[DHAnderson]

@CharlieCrackle

The uplink port is set to allow VLAN 1 or all VLANS?

[CharlieCrackle]

Just an update for you all.  it has taken along time for support to get back to me.

Switches boot with an IP address 1.1.1.1   it then changes to the static address or the DHCP address.

Support can not tell me why they use 1.1.1.1  (some one else IP address) or how the switch could ever use this address to get to the cloud.   If you are upgrading 3 switches at a time   then they will all reboot and at the same time there will be 3 devices with 1.1.1.1  causing the MX to error with duplicate IP.

If some one from engineering could explain what the rational is it would be great.

[CharlieCrackle]

no both switches are set to DHCP.   it has been 9 hours now and they are still doing this.  what is wierd is all the cameras and AP that are connected to the switch are showing as not connected to cloud also but are working ok.  and the users on the switch are still working ok.  I don't get it at all and at this point support don't either.  Some one is going to site to power cycle the switches.

[Gareth_]

Charlie did you ever get to the bottom of this? We have MS120's with this behaviour.

[CharlieCrackle]

No I only know they boot with 1.1.1.1   address  no one has explained the reason.  I just upgrade 1 switch at a time now

[Gareth_]

Thanks Charlie. Just supporting your information adding that our switches have this behaviour also. Whilst we were resolving another issue regarding a DNS change in our network it is something that baffled us when it appeared in our logs.