Catalyst Cloud Monitoring Offboarding Feedback

[FCU_JE]

I'm investigating the transition from "Cloud Monitoring" to "Cloud-Native IOS XE" (Hybrid Operating Mode).

I took one of my stacks running in cloud monitoring and went through the removal steps in the dashboard. Easy enough by itself but on further investigation, I think Meraki is failing to completely clean up the configurations it makes as part of onboarding.

Below are what I found which are configurations Meraki applies to switches/stacks but DOES NOT remove as part of offboarding. In no particular order:

aaa authentication login MERAKI local
aaa authorization exec MERAKI local
!
lldp run
!
ip ssh server algorithm authentication publickey password keyboard
!
line vty 16 17
 transport input ssh
line vty 18 19
 access-class MERAKI_VTY_IN in
 access-class MERAKI_VTY_OUT out
 authorization exec MERAKI
 login authentication MERAKI
 rotary 50
 transport input ssh
!
netconf-yang
!

Now in fairness, there could be a few of those which might make sense to leave behind as they could hurt an admin's ability to troubleshoot/manage the switch/stack if it's already functioning (ip ssh and lldp) but as for aaa/netconf/the line configs? Those don't need to persist post-offboard. Below is some show log output if it helps show what *was* removed/done.

switch#show logg | include HA_EM-6
May  6 20:08:19.107: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: !Start running
May  6 20:08:19.109: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: config terminal lock
May  6 20:08:19.223: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: !Removing brownfield device config
May  6 20:08:19.223: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no snmp-server enable traps smart-license
May  6 20:08:19.550: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no snmp-server enable traps config-copy
May  6 20:08:19.661: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no snmp-server enable traps config-ctid
May  6 20:08:19.774: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no snmp-server enable traps config
May  6 20:08:19.886: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1024
May  6 20:08:20.097: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1030
May  6 20:08:20.209: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1031
May  6 20:08:20.322: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1001
May  6 20:08:20.433: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1002
May  6 20:08:20.546: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1003
May  6 20:08:20.656: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1004
May  6 20:08:20.770: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1007
May  6 20:08:20.882: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 2002
May  6 20:08:20.993: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1011
May  6 20:08:21.106: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1012
May  6 20:08:21.218: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1013
May  6 20:08:21.329: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1014
May  6 20:08:21.441: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1015
May  6 20:08:21.555: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1016
May  6 20:08:21.665: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1017
May  6 20:08:21.778: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1018
May  6 20:08:21.894: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1020
May  6 20:08:22.006: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry ietf subscription 1021
May  6 20:08:22.118: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry transform MERAKI_INTF_STATS_DELTA
May  6 20:08:22.330: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no telemetry transform MERAKI_PORTCHANNEL_STATS_DELTA
May  6 20:08:22.645: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no device-tracking policy MERAKI_POLICY
May  6 20:08:23.086: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no snmp-server host 18.232.244.158 traps version 2c public
May  6 20:08:23.297: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no logging host 18.232.244.158
May  6 20:08:23.416: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no ip route 18.232.244.158 255.255.255.255 Null0
May  6 20:08:23.629: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: !Removing tls config
May  6 20:08:23.630: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no crypto tls-tunnel MERAKI-PRIMARY
May  6 20:08:24.093: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no crypto pki trustpoint MERAKI_TLSGW_CA
May  6 20:08:24.206: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: y
May  6 20:08:24.425: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: % Be sure to ask the CA administrator to revoke your certificates.
May  6 20:08:24.425: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: !Removing user config
May  6 20:08:24.426: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no username meraki-user
May  6 20:08:24.539: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: y
May  6 20:08:24.758: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: ip ssh pubkey-chain
May  6 20:08:24.869: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no username meraki-user
May  6 20:08:24.982: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: exit
May  6 20:08:25.094: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no ip ssh port 2222 rotary 50
May  6 20:08:25.206: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no ip access-list extended MERAKI_VTY_IN
May  6 20:08:25.654: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no ip access-list extended MERAKI_VTY_OUT
May  6 20:08:25.767: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no interface Loopback1000
May  6 20:08:26.333: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: !Clearing VTY lines
May  6 20:08:26.333: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: do-exec clear line 16
May  6 20:08:26.445: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: y
May  6 20:08:26.556: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: do-exec clear line 17
May  6 20:08:26.670: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: y
May  6 20:08:26.780: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: !Removing VTY config
May  6 20:08:26.781: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: line vty 16 17
May  6 20:08:26.894: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no rotary 50
May  6 20:08:27.005: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no access-class MERAKI_VTY_IN in
May  6 20:08:27.117: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no access-class MERAKI_VTY_OUT out
May  6 20:08:27.229: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no authorization exec MERAKI
May  6 20:08:27.342: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no login authentication MERAKI
May  6 20:08:27.454: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: exit
May  6 20:08:27.565: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: no event manager applet MERAKI-DASHBOARD-CLEANUP
May  6 20:08:27.776: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: end
May  6 20:08:27.789: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: write memory
May  6 20:08:32.812: %HA_EM-6-LOG: MERAKI-DASHBOARD-CLEANUP: !Finish running
switch#

[alemabrahao]

You can try a Manual Cleanup, but I would recommend contacting Meraki support.

[FCU_JE]

Not really the point though. If Meraki installs a config during onboarding it should (subject to reasonable exceptions) remove it during offboarding.

[Brash]

That's an interesting one.

I would have assumed it would remove this config too as once the TLS tunnel is brought down there's need to keep the VTY and AAA config there.

Worth raising with Meraki support to validate the expected behaviour.

[Tony-Sydney-AU]

Hi @FCU_JE ! Hope you're doing great.

Thanks for bringing this behaviour to our attention.

I'm checking with Internal Team to validate if this is an expected behaviour.

But feel free to open a support case if you are in a hurry. I'm based in Australia so my time zone isn't helpful. Maybe you can find your answer faster.

I'll keep you updated with my findings anyway.

[FCU_JE]

I'm not in a hurry on this one yet. I made this as a post vs a support case mainly to create awareness and I figured it might get more visibility/treated as a bug rather than the support approach (which I've noted is slower than I've been used to with Meraki in previous years).

Plus by sharing broadly, maybe people will try to reproduce it to see if they see the same thing.

I'm still working out my process for deployment of new 9200L switches so I have the luxury of time to test the new cloud-native hybrid mode to see if it will work for us (and ultimately save me time later/post-deployment).

[Tony-Sydney-AU]

Hi @FCU_JE ,

Yes, I firmly believe that our Meraki Community here is a better starting point than opening a support case.

And yes, I confirmed that behaviour is unexpected and you need to raise a ticket. So, as you said you have the luxury of time, please open a case with us at Meraki Support.

The good news is this behaviour was already reported before and our teams are working on fixing it.

But I must ask you to please open support case and tell me the case number over here. This would allow me to give more visibility to this issue here internally.

I don't know many details about how or what triggers this issue. But in essence, when you remove a Catalyst switch that was operating in Monitoring mode, then the dashboard pushes and the switch runs a removal script. You can read more about it here.

Usually this script runs smoothly but, as you already know, sometimes there are things left over.

So again, if you open a support case and tell me the number I'll be able to go ahead and give more visibility to Internal Teams.

Thank you one more time!

[FCU_JE]

I'm still in rapid testing mode and in hindsight I may have exaggerated just how much luxury I have with regard to timeline.

I will skip over the case for now but my original post should give anyone in Meraki the ability to try and reproduce the problem. It's as simple as onboard a switch, offboard a switch, then observe the config post-offboard.

[br_bass_fishin]

My team and I have been on-boarding 9200 and 9300 stacks for a client and per their request, they are wanting them brought into the Meraki dashboard under the hybrid mode. So far all the 9300 stacks we've deployed have run smooth.  From configs to bringing them into the Meraki dashboard and the tunnels staying up while the on-boarding process happens. We can launch the CLI and run commands through the dashboard without any issues. However, with that said, the 9200's are the devil.  They are configured identically to the 9300's and for the past 3 weeks and about a 100 emails back and fourth with Meraki and Cisco, no one can explain why the 9200's will not populate in the dashboard.  Even better, The dashboard says they are online, but the local ip address for this stack are wrong and the gateway doesnt exist, but if I launch the terminal (even though I shouldn't be able too because they haven't connected to the dashboard) I can do a "Show Run" command and low and behold the correct config with ip address and routes are on the switch.  I've removed and added this stack back 3 time per Meraki/Cisco tech's and so far nothing has worked.  This stack is in a production environment and has been since we deployed it, but its never communicated with the Dashboard, but like I said earlier the 9300 stacks have never had an issue communicating.  Any thoughts?

[FCU_JE]

This will probably not be the productive/utilitarian response you need.

Purely 9200L units for us. I think all I can say with regard to the Hybrid mode is that it is incredibly disappointing. IMO Cisco/Meraki engineering didn't actually test this new management model before rolling it out and deciding to deprecate the old method.

While I like the change of not needing a separate onboarding application, the problem I have is that the new onboarding process is *really* bad for switch stacks. Not to mention that if you need to replace a stack member (or expand/shrink a stack for that matter) it's not clear how you are meant to do that as the administrator with hybrid mode. With the cloud monitoring model, I had tested this and found that you simply remove the old switch, put in the new one, and .... Merkai catches up automatically. No additional action required.

From what Meraki support has told me, you need to basically do onboarding of each individual stack member to Meraki which to me sounds insane.

[bradly]

Hi @Tony-Sydney-AU 
Do you know if this issue was ever resolved? Or do we need to open a support case if we are experiencing this issue as well? For us, the AAA configuration, and vty lines are being missed by the Meraki cleanup script.

[FCU_JE]

Snarky and unsolicited response: If you have to ask Meraki if something was fixed, it never was, and they don't have an intention to fix it.

I have several outstanding issues with Meraki. They appear to have zero incentive to actually fix product/service issues.

[bradly]

I am getting the vibe from Meraki that they don't want anything to do with the legacy Cloud Monitoring service anymore. I've raised a couple of support cases, and the general response is "Yep, legacy cloud monitoring is broken. We are turning it off the 31st of March, please upgrade to Cloud Management" 

Anyway, I fixed it myself by putting together a little command script that I copy and paste, and it manually goes through and cleans up all the left-over Meraki configuration. 

Funny thing is, I actually thought the configuration that was left behind was intentionally left behind haha. I never thought it was an issue until a saw this thread. 

[Tony-Sydney-AU]

Hi @bradly and @FCU_JE 

Thanks for bringing this up, @bradly .

Does it happen all the time you offboard a switch? How often does it happen?

Answering your question, @bradly : no it wasn't.

@FCU_JE , I honestly share your frustration. From a customer perspective, it might feel like Support doesn't want to fix it. But I assure you it's absolutely not the case.

@bradly , If you confirm that this happens fairly often, then there might be a good chance We can identify a pattern and work from there. If that's the case, We would need you to create a support case and We would work with internal teams on that.

My understanding is this off-boarding issue has been an edge case that only happened like one in a million times and it's very hard to reproduce. Just so you have an idea, I've been working in support for quite a long time now; to be fair, I only saw it happening once.

Like I mentioned in my previous response, this behaviour was already reported and our Internal Teams Investigated it. Now that you asked here, @bradly , I went ahead and checked  that case again. I found my colleague did a repro lab and it didn't happen in a lab environment. Customer who opened that case closed it and did just like you did: moved on and removed configs manually.

As you know, When you off-board from Cloud Monitoring, dashboard pushes an EEM script that your switch runs after it goes off-line and detached/disassociated from your network. This document describes it. What causes this script to fail still unclear.

In my experience over the years in IT: by the end of the day, as large as Internal Teams can be, they are a limited resource and there are other issues to investigate. In addition, they also have to keep developing new features and working on Feature Requests. Every tech company have this limitation.

If you really want to move ahead and investigate this issue, maybe support isn't the best path in this case. It's definitely not like they don't want to work in this issue. They know the limitation above so actually they are saving your time.

Perhaps you would get better results if you discuss this with your Sales Account Manager. If you can expose your use-case and pain-points, they might get to Internal Teams to investigate more.

Hope you can get more traction by discussing with your Sales Account Manager. However, keep in mind that even if you engage your Sales Account Manager, I suspect it isn't going to be a quick solution. That's because you have a workaround and this is a minor impact (IMO).

[bradly]

Hey @Tony-Sydney-AU

I must be the luckiest person in the world! It's happened to all 130 Catalyst switches I've migrated so far 😂 I should probably get some Lotto tickets for tomorrow night 😁

Yeah, the EEM script has failed to remove the VTY lines and AAA configuration every single time, requiring me to run through it with my mini script. It hasn't completed successfully on every C9300 and C9500 switch I've migrated so far. That's why I started thinking it was intentional. 

Nah, I am not going to raise a case for this. It's no impact, and we have a quick and easy workaround for it like you said. So, we are just moving forwards, getting off legacy Cloud Monitoring and on to Cloud Management and moving on. 

And completely understand the Meraki team only has finite resources and has to prioritise their workload, and with Cloud Monitoring going away in about two months, I understand it is not at the top of their list to troubleshoot weird issues. Especially issues that aren't disrupting services or causing outages. 

I just wanted to check and see if there was a solution (now that I know it's an issue), and maybe it was an easy solution, or I was doing something wrong. 

Thank you @Tony-Sydney-AU @FCU_JE for your input 😊

[Tony-Sydney-AU]

Oh my God, @bradly!!! 😮 Really??

Now you got me curious about it. What was their IOS XE version? Were they all running the same version?

[bradly]

They are all running IOS XE 17.9.5. They were onboarded into Meraki Monitoring around May 2024 using the legacy monitoring onboarding process. They were onboarded at version 17.9.5. 

Can I send you a sanitised copy of our configuration? If you're interested of course. 

[Tony-Sydney-AU]

Yes, please! I'll try to find some pattern.

[FCU_JE]

Trying to reply to a lot of the back and forth here (but succinctly, so details won't be accurate):

• "From a customer perspective, it might feel like Support doesn't want to fix it. But I assure you it's absolutely not the case."
  • I like to assume goodwill too, but I found this issue Q1 or Q2 last year and I've had no updates on it.
• "My understanding is this off-boarding issue has been an edge case that only happened like one in a million times and it's very hard to reproduce"
  • Absolutely not the case for me. All my memories are of course faint now but this was *VERY* consistent in my testing.
• "What causes this script to fail still unclear."
  • But what is clear is that the Meraki R&D/devs didn't actually test it. Otherwise they would've caught it. It's embarrassing.
• "they are a limited resource and there are other issues to investigate"
  • Agreed, but if you want to (continue to) sell the service/product (in the interest of retaining Meraki's place in the market and as a result, ensure your own self-preservation), it's got to be stable.
• "They know the limitation above so actually they are saving your time"
  • Except (last I checked) these limitations aren't even documented in the Meraki docs.
• "Now you got me curious about it. What was their IOS XE version? Were they all running the same version?"
  • Not who you asked, but I was testing on a few different versions at the time of 17.12 and 17.15 and it was all consistently not removing the configs. Again, no one is testing this (chose your expletive).
• I don't have 130 switches like the other commentor. Order of magnitude smaller.

All these issues with the Meraki ... whatever you're calling it ... monitoring for Catalyst while not large individually are adding up and make me MUCH less eager to recommend my managers to consider Meraki in the future.

The only things that really separate networking vendors these days are security reputation, support quality, the management plane, and cost. Meraki (Cisco) is not leaps and bounds ahead of the pack on any of these metrics.

[Tony-Sydney-AU]

Thank you, @FCU_JE ! Can you please send me your support case number over a private message?

I'll have a look at your case.

And I need to quote you on this:

• "Agreed, but if you want to (continue to) sell the service/product (in the interest of retaining Meraki's place in the market and as a result, ensure your own self-preservation), it's got to be stable."

100% agree with you. I believe our community is a good space not just to vent our frustrations, but also to do bring your feedback to internal teams and work on it.

That's why I ask the support ticket number.

Let me take a look and I'll update you back. 

[FCU_JE]

In hindsight I think I misspoke/was mistaken. I don't think I ever made a support ticket for *this* issue. I think at the time I created this community post I was so fed up with the (lack of) support I decided to give the community post route a try instead.