Catalyst Cloud Monitoring Offboarding Feedback

[FCU_JE]

I'm investigating the transition from "Cloud Monitoring" to "Cloud-Native IOS XE" (Hybrid Operating Mode).

I took one of my stacks running in cloud monitoring and went through the removal steps in the dashboard. Easy enough by itself but on further investigation, I think Meraki is failing to completely clean up the configurations it makes as part of onboarding.

Below are what I found which are configurations Meraki applies to switches/stacks but DOES NOT remove as part of offboarding. In no particular order:

aaa authentication login MERAKI local
aaa authorization exec MERAKI local
!
lldp run
!
ip ssh server algorithm authentication publickey password keyboard
!
line vty 16 17
 transport input ssh
line vty 18 19
 access-class MERAKI_VTY_IN in
 access-class MERAKI_VTY_OUT out
 authorization exec MERAKI
 login authentication MERAKI
 rotary 50
 transport input ssh
!
netconf-yang
!

Now in fairness, there could be a few of those which might make sense to leave behind as they could hurt an admin's ability to troubleshoot/manage the switch/stack if it's already functioning (ip ssh and lldp) but as for aaa/netconf/the line configs? Those don't need to persist post-offboard. Below is some show log output if it helps show what *was* removed/done.

switch#show logg | include HA_EM-6
May  6 20:08:19.107: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: !Start running
May  6 20:08:19.109: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: config terminal lock
May  6 20:08:19.223: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: !Removing brownfield device config
May  6 20:08:19.223: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no snmp-server enable traps smart-license
May  6 20:08:19.550: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no snmp-server enable traps config-copy
May  6 20:08:19.661: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no snmp-server enable traps config-ctid
May  6 20:08:19.774: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no snmp-server enable traps config
May  6 20:08:19.886: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1024
May  6 20:08:20.097: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1030
May  6 20:08:20.209: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1031
May  6 20:08:20.322: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1001
May  6 20:08:20.433: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1002
May  6 20:08:20.546: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1003
May  6 20:08:20.656: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1004
May  6 20:08:20.770: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1007
May  6 20:08:20.882: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 2002
May  6 20:08:20.993: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1011
May  6 20:08:21.106: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1012
May  6 20:08:21.218: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1013
May  6 20:08:21.329: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1014
May  6 20:08:21.441: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1015
May  6 20:08:21.555: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1016
May  6 20:08:21.665: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1017
May  6 20:08:21.778: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1018
May  6 20:08:21.894: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1020
May  6 20:08:22.006: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1021
May  6 20:08:22.118: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry transform MERAKI_INTF_STATS_DELTA
May  6 20:08:22.330: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry transform MERAKI_PORTCHANNEL_STATS_DELTA
May  6 20:08:22.645: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no device-tracking policy MERAKI_POLICY
May  6 20:08:23.086: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no snmp-server host 18.232.244.158 traps version 2c public
May  6 20:08:23.297: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no logging host 18.232.244.158
May  6 20:08:23.416: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no ip route 18.232.244.158 255.255.255.255 Null0
May  6 20:08:23.629: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: !Removing tls config
May  6 20:08:23.630: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no crypto tls-tunnel MERAKI-PRIMARY
May  6 20:08:24.093: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no crypto pki trustpoint MERAKI_TLSGW_CA
May  6 20:08:24.206: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: y
May  6 20:08:24.425: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: % Be sure to ask the CA administrator to revoke your certificates.
May  6 20:08:24.425: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: !Removing user config
May  6 20:08:24.426: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no username meraki-user
May  6 20:08:24.539: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: y
May  6 20:08:24.758: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: ip ssh pubkey-chain
May  6 20:08:24.869: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no username meraki-user
May  6 20:08:24.982: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: exit
May  6 20:08:25.094: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no ip ssh port 2222 rotary 50
May  6 20:08:25.206: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no ip access-list extended MERAKI_VTY_IN
May  6 20:08:25.654: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no ip access-list extended MERAKI_VTY_OUT
May  6 20:08:25.767: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no interface Loopback1000
May  6 20:08:26.333: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: !Clearing VTY lines
May  6 20:08:26.333: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: do-exec clear line 16
May  6 20:08:26.445: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: y
May  6 20:08:26.556: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: do-exec clear line 17
May  6 20:08:26.670: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: y
May  6 20:08:26.780: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: !Removing VTY config
May  6 20:08:26.781: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: line vty 16 17
May  6 20:08:26.894: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no rotary 50
May  6 20:08:27.005: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no access-class MERAKI_VTY_IN in
May  6 20:08:27.117: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no access-class MERAKI_VTY_OUT out
May  6 20:08:27.229: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no authorization exec MERAKI
May  6 20:08:27.342: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no login authentication MERAKI
May  6 20:08:27.454: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: exit
May  6 20:08:27.565: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no event manager applet MERAKI-DASHBOARD-CLEANUP
May  6 20:08:27.776: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: end
May  6 20:08:27.789: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: write memory
May  6 20:08:32.812: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: !Finish running
switch#

[alemabrahao]

You can try a Manual Cleanup, but I would recommend contacting Meraki support.

[FCU_JE]

Not really the point though. If Meraki installs a config during onboarding it should (subject to reasonable exceptions) remove it during offboarding.

[Brash]

That's an interesting one.

I would have assumed it would remove this config too as once the TLS tunnel is brought down there's need to keep the VTY and AAA config there.

Worth raising with Meraki support to validate the expected behaviour.

[Tony-Sydney-AU]

Hi @FCU_JE ! Hope you're doing great.

Thanks for bringing this behaviour to our attention.

I'm checking with Internal Team to validate if this is an expected behaviour.

But feel free to open a support case if you are in a hurry. I'm based in Australia so my time zone isn't helpful. Maybe you can find your answer faster.

I'll keep you updated with my findings anyway.

[FCU_JE]

I'm not in a hurry on this one yet. I made this as a post vs a support case mainly to create awareness and I figured it might get more visibility/treated as a bug rather than the support approach (which I've noted is slower than I've been used to with Meraki in previous years).

Plus by sharing broadly, maybe people will try to reproduce it to see if they see the same thing.

I'm still working out my process for deployment of new 9200L switches so I have the luxury of time to test the new cloud-native hybrid mode to see if it will work for us (and ultimately save me time later/post-deployment).

[Tony-Sydney-AU]

Hi @FCU_JE ,

Yes, I firmly believe that our Meraki Community here is a better starting point than opening a support case.

And yes, I confirmed that behaviour is unexpected and you need to raise a ticket. So, as you said you have the luxury of time, please open a case with us at Meraki Support.

The good news is this behaviour was already reported before and our teams are working on fixing it.

But I must ask you to please open support case and tell me the case number over here. This would allow me to give more visibility to this issue here internally.

I don't know many details about how or what triggers this issue. But in essence, when you remove a Catalyst switch that was operating in Monitoring mode, then the dashboard pushes and the switch runs a removal script. You can read more about it here.

Usually this script runs smoothly but, as you already know, sometimes there are things left over.

So again, if you open a support case and tell me the number I'll be able to go ahead and give more visibility to Internal Teams.

Thank you one more time!

[FCU_JE]

I'm still in rapid testing mode and in hindsight I may have exaggerated just how much luxury I have with regard to timeline.

I will skip over the case for now but my original post should give anyone in Meraki the ability to try and reproduce the problem. It's as simple as onboard a switch, offboard a switch, then observe the config post-offboard.