Meraki shut down my entire network for "non payment" of 2 devices

[jmorphew]

We have a data center, main office, and remote office.  All network equipment is Meraki, around 75 devices in total. We've been customers for 3-4 years and have so far been very happy with the products.  Last month I worked with a local company to install two smaller Meraki switches at a remote site warehouse.  Adding the license for these two devices was overlooked.  Today, without notice, Meraki shut down our entire network.  All three sites were taken offline, including our data center. 

I was not able to login to the portal because we use SSO/Duo, but our local support tech was.  He said we were shut down for non payment.  All devices had a 5 year agreement, just the two new switches were not added after installation.  Meraki support gave us a "grace period" to get the new license added and brought us back online.  I was not notified in the dashboard or by email, in advance or at the time of the shutdown.  To my knowledge, the only way to see a warning was through the licensing page in the dashboard.

Why would Meraki shut down our entire network, without notice, for non compliance of two devices?  How is this even legal?  They shut down over 75 paid for and supported devices.  This had a huge impact on our business for the day. I'm very frustrated and worried about this happening again in the future.

[PhilipDAth]

When you add new non-licenced devices you get a 30 day grace period.  During that grace period, Meraki do send email notices, constantly.  They typically go to all "Meraki" (not SAML) administrator accounts.  You need to make sure those accounts are monitored.

Alternatively, you can specify an email address for them to go to under Organisation/Settings.

So what happened is you used up your first grace period, didn't monitor the accounts configured for the warning emails (you would have gotten them daily), and once that grace period had run out, they shut you down.  Fortunately, they have given you a second grace period.

This is no different to Microsoft Office 365, Microsoft Azure, Amazon AWS, ...

All of these providers send the warnings to the configured email addresses.  If you ignore them, they shut you down after a grace period.

[jmorphew]

Respectfully, there is a huge difference between shutting down 2 non-compliant devices and an entire network.  They shut down paid for and supported devices with valid licenses.  So if I have a camera or environment monitor go noncompliant next, they’re going to shut down my datacenter?

I am an admin, and I really don’t see any email alerts.  I see plenty of normal outage alerts from Meraki, but nothing from licensing.  Any advice on what those emails looks like would be appreciated.  Are you saying that because my account uses SSO with Azure, that they don’t send me the licensing notifications?

[jmorphew]

I think I see what you are referring to now.  We use SAML administrator roles for all internal staff.  The privilege assigned to my group is organization admin.  The licensing alerts are set to go to all organizational and network admins.  But because I'm in a group, Meraki doesn't include me in these alerts?  How is anyone expected to know that?

This is a failure on Meraki's part anyway I look at it.  It was a small oversight from the network tech that Meraki escalated into a networkwide outage.  They can't send me a personal email before shutting down several hundred thousand dollars worth of paid for hardware?  Very frustrated.

[ww]

You can look into moving to pdl lic

https://documentation.meraki.com/General_Administration/Licensing/Meraki_Per-Device_Licensing_Overvi...

[jmorphew]

Do you know if per device licensing prevents Meraki from shutting everything down when one device is not compliant?

[Ryan_Miles]

Covered in the last bullet here

[jmorphew]

Thank you, I will also be switching to per device licensing. 

[Ryan_Miles]

Philip is spot on with everything he mentioned above. I will add a few points as well. My intent is not to place any blame. Rather just hoping to fill in some areas that might not be fully understood and hopefully it can help anyone that lands on this thread in the future. Also, it might give you some ideas on how to improve some areas of config in your org.

This doc covers how SAML/SSO and email alerts work. It matches what Philip stated about needing at least one local admin account.

As soon as licensing is oversubscribed it will place an org into an out of compliance/warning state and initiate the 30 day grace period. Also, devices only consume a license when placed in a network. Unused devices in the org inventory don't consume a license.

I do also see your org has 8 local admins with write or read access. All of them appear to be partners/providers based on their email domains. They should have all been receiving these emails multiple times during the 30 day grace period (at 30, 21, 14, 7, and 1 days). Based on how your admin list is set up it appears no employees are local admins and therefore no one at your company would have received a license alert email.

Another thing I noticed. The missing license that caused the compliance issue was emailed to you. However, the email it was sent to is different from what I see in your SAML login history. So, I'm not totally sure if you received the original order email with the license that should have been claimed back in November. Orders going to bad email addresses happens quite a lot as it depends on the partner/reseller that placed the order entering a legit address. I see typos at times or many times it's sent to someone outside of IT that will receive the email, but might have no idea what it is (like a Finance person, branch manager, etc).

That reminds me of one more point. You should always be adding gear to orgs via the order number. That brings in the serials and license keys. In a case of just serials being added it can lead to this exact problem. I probably spend an hour or two every single week helping customers clean this up and it can all be avoided by claiming the order number to begin with.

Bottom line, best practice is using SAML/SSO is totally fine. Just make sure you have at least one local admin configured with an email address (could be a mailing list) that people pay attention to. Having all your local admin accounts belonging to an external company isn't something I'd recommend (unless it's Meraki as a service and you don't own the hardware).

[jmorphew]

Thanks for the reply, Ryan.  We will be getting rid of SAML at the first opportunity.  I receive other Meraki notifications all the time with my current account, I don’t see why licensing notifications should be any different.

Yes, I had the license from an order confirmation sent from our support partner.  I rely on partners for setup and support, and the tech clearly forgot to install the license.  I doubt they watch the email alerts very closely either, they probably support hundreds of companies.  Mistakes happen, a missing license key on two switches feels like a small mistake to me.  It did not need to escalate like this.

Frankly, I don’t see how you can legally shut down our entire network for noncompliance of two minor devices.  I will really struggle to put any additional Meraki equipment in our data center knowing that it can be disabled so easily.

Why is this your policy?  Send me an email, give me a call, something!  I am not trying to cheat Meraki out of any money here.  There is no reason to completely lock us out of a network when 1% of it is missing a license key.  I didn’t even realize you were capable of shutting down our local, fully paid for hardware.  I would revisit this policy, it is terrible customer service and unlike anything else I’ve experienced with Meraki. 

[Ryan_Miles]

If you look under Network-wide > Configure > Alerts you'll see your default alert recipients are you, a coworker of yours, and All network admins (meaning all those 8 partner admins you have defined). Any email address can be typed in there even if they aren't admins on the admin page. That's why you get the general network alerts.

[jmorphew]

Thanks Ryan, I do appreciate the technical explanation of how this went wrong and the fast responses from everyone.  I now have a good idea of how to prevent this in the future.

I would like to hear a company statement on why Meraki has this policy?  Meraki sent some automated emails, then shut down our entire network for a minor infringement.  This had a significant impact our company for the day and we deserve an explanation of why this is your business practice.  

[Ryan_Miles]

In dashboard go to Organization > Configure > License info. On that page it shows who your Meraki Sales rep is. I would contact them to further discuss Meraki license policies.

[jmorphew]

I already contacted my account rep.  He apologized for the outage and said he does not agree with the policy.  What's the next escalation?  

[PhilipDAth]

@jmorphew - you are trying to apportion blame on others when it is your company's process that was at fault.

First, if your company's suppliers were listed as administrators, they would have been getting the warning emails.  We are a Cisco Meraki reseller, and we act on every licence and shutdown notification we receive on behalf of one of our customers.  Everyone of them.

I can't understand why a Cisco Partner would choose to ignore such notifications, knowing what would happen to their customer.

Perhaps you could start by asking those suppliers why they ignored the warnings, why they didn't reach out to you and take care of you.

Who originally setup your Cisco Meraki network?  They should have known to setup the notifications to a monitored email address.  It's not just licence notifications, but other critical notices (like security notificatoins).  Someone should be reviewing them.

Cisco Meraki clearly spell out the terms and conditions (such as in the FAQ posted above).  They send constant notifications to all the registered Meraki account admins.

I really can't think of what more Meraki could have done here.  They did exactly what their FAQ says they do.  They did what their T&Cs say.

Their is nothing to escalate.  The Cisco Meraki process worked exactly as intended.

At the end of the day - it was your company that made the mistake.  Loading in devices rather than the order number, only having notifications being sent to people that ignored them.

It is your process that needs to change.

I understand you are feeling pretty savage about what happened.  You had a complete network outage due to several humans making a mistake.  And that is what it was - a human mistake.

And it wasn't even just one failure - it required all of those people getting the warnings to ignore them.  If only one of those people had flagged it with you, this issue would not have happened.

You can keep using SAML.  That is a good approach.  You just need to configure the notifications to be sent to a monitored email, talk to your suppliers and ask them to never ignore such emails again, and move on.

I'm sorry for what happened to you - but this is the cold reality.

[jmorphew]

Yes, mistakes were clearly made on our end I and did not deny that.  We are a smaller company, and internal staff cannot be experts in every single product we have.  That’s why Meraki is so appealing, because it’s easy to use.  I was very impressed with the vendors I worked with on the two switches, but clearly they made a mistake too.  And yes, the vendor that helped us setup SAML over overlooked a notification setting.

But I strongly disagree that Meraki is blameless.  Their policy is horrible and should be changed.  If a single human at Meraki had made an effort to reach out to me personally, this would have been avoided.  If they had posted a warning in the dashboard, this would have been avoided.  There are a ton of ways Meraki could make this a better experience.  Over the years, I have had a license or two lapse in coverage.  I have never had a company remove my access to hardware that is paid for and under a support contract.  I have also never had a company cause a network wide outage because of a minor billing discrepancy. 

Like many companies, we are not staffed to understand every nuance of the products we use.  We rely on solid partners to help us with our infrastructure.   In my opinion, Meraki failed as a partner today.

[jmorphew]

And most importantly, if they had just removed access to the two devices with licensing problems I would not be upset.  That would be a reasonable response.  Removing functionality from completely unrelated devices is not acceptable.  I do think this should be escalated, awareness should be spread, and Cisco should respond with why they have this very extreme policy.

[AndrewCarey]

I'd also point out the Intro to the Meraki Platform on-demand elearning. The licensing module covers exactly this scenario, and the trade offs between the two options. 

[jmorphew]

I’ll add something for any small/medium business reading this.  We enabled SAML because we thought it made us more secure, especially having duo for two factor auth.  After this event, I don’t think any small or medium business should use SAML.  It added unnecessary complexity to the notification system (we can’t be the only ones that overlooked that confusing checkbox). 

But most importantly, when Merkai decided to lock us out of our entire network, they effectively killed SAML auth.  So we were no only locked out of our network, but also the Meraki dashboard.  I now understand you’re supposed to have break glass Meraki cloud accounts, but for a company with two admins that pretty much defeats the purpose of SAML.

[jmorphew · Accepted solution]

Thanks again to everyone that helped me understand the technical failures on this issue.  I met with Meraki today.  They confirmed everything in this forum.  They basically said this is their company policy and they don’t plan on changing it.  It’s their way of making sure customers don’t have non licensed hardware.  Still seems ridiculously heavy handed towards the small/medium businesses that are honest.  It has shaken my faith in a company that I really liked.  Large enterprises are exempt because they spend enough money to get Enterprise Agreements with much more associated leniency and customer support.  I wish they would do this for all customers.

To resolve the issue, we will be moving to per device licensing.  I don’t like the idea of managing 125 licenses and renewals, but it’s better than the risk getting shut down again.  We will also be moving back to cloud authentication to simplify the alerts and avoid being locked out of the dashboard again.

[Jeff_Larkham]

This just happened to my company today.  We had one AP over the limit and they shut down 4 office networks. We were never notified!  The mere fact that Meraki thinks it is OK to turn off a company's network is unbelievable.  Maybe for some companies it is not a big deal, but we deal with things that affect people's lives.  When we decided to use Meraki we trusted they would do the right thing.  That trust is gone.  If knew this before we decided on Meraki we would never have used them.  With great power comes great responsibility and they obviously don't take that responsibility seriously.  I'll never trust them again and make sure to let everyone I know what happen.    

[jmorphew]

It's been a year and a half, and I still get upset thinking about it!  I also tell everyone that will listen.  Hopefully enough people take notice that Meraki will change their business practices.  They could easily disable unlicensed devices without impacting the rest of the network.  They could also have the account rep reach out before taking such drastic measures.  I really hope they change this policy!