Stable Firmware VS. Beta Firmware

[JPAWELCHAK]

Dear Meraki,

Don't get me wrong, we love your products and we have both sold and deployed a HUGE number of Meraki MX, MS and MR devices to our customers. That being said, however, we would very much like a MX Product Manager or Support Engineer please explain to us why it is Meraki thinks it is acceptable to push what you yourselves call "Beta Firmware" to production environments? My counterparts in the Meraki community that I have discussed this with are all frustrated with your firmware release schedule and why your Support Engineers request that we upgrade to beta when according to the firmware release notes the firmware does not address the issue at hand. Even more frustrating is the fact that the last stable update for the MX64 (I don't have time to look for other examples) was 12/2016 and there are vulnerabilities that have closed in the beta version of your firmware and no stable releases. Yes, you read that right, last "Stable Firmware" for the MX64 was 12/2016 and the best I can tell there have been 24 updates to your beta release and ZERO STABLE UPDATES RELEASED SINCE 12/2016!  What are we paying a premium for Meraki licenses especially for the Advanced Security License (we never deploy the Enterprise License) for if Meraki isn't releasing stable firmware updates in a timely fashion? I thought Meraki had a quarterly firmware release schedule or did I dream reading this OR is Meraki failing to keep up with said release schedule? Being that the MX64 has been neglected am I to assume other MX device firmware has also been neglected? If so some communication, transparency and direction is in order and would be appreciated.

Finally, why are we all suddenly being inundated with alerts regarding beta firmware updates being available for all the networks we manage? Best we can tell it appears in order to gain access to any of the firmware that closes numerous security vulnerabilities we have to go to firmware yet when we challenge Meraki support Engineers on this the claim it is stable firmware. If its stable firmware call it stable if it's beta call it beta. Which is it, you cant have it both ways?

[PhilipDAth]

I agree entirely!!!

Stop blasting us about beta firmware updates and give us more timely production releases.

[MilesMeraki]

This is a valid point. However what's to say that other vendors don't the same however mark the release as being "Stable" even though it's very much a beta release? 

[PhilipDAth]

It doesn't matter what other vendors do when you are using Meraki kit ...

[JPAWELCHAK]

---

@MilesMeraki wrote:

This is a valid point. However what's to say that other vendors don't the same however mark the release as being "Stable" even though it's very much a beta release? 

---

Understood but I'm only concerned what Meraki does/does not do. Cisco Meraki is the only network stack we sell, deploy and manage. They don't get to call it Beta and then claim its Stable during a support call. Just be more transparent and communicate this matter better but their recommendation of updating to beta firmware goes against decades of experience why one should not update to "beta" releases of anything. We support a Stock Exchange and a leader in event ticketing and it doesn't go over well with their IT departments when they are getting alerts from their Meraki Dashboard encouraging them to instal beta firmware or that we are recommending (unwillingly) to instal beta firmware not to mention their IT policies stating it is not permitted.

[Dashboard_DJ]

@JPAWELCHAK I definitely get where you're coming from here. The current MX GA is long in the tooth and many great enhancements and bug fixes have been available in wired13 for some time now.

A couple thoughts here:

• The Meraki product teams recently made significant changes to the firmware release process to provide more structure, provide better QA, and also change the name on late-stage firmware from beta to stable release candidate (a label that more accurately reflects the code)
• Introducing stable RC step added additional bake time requirements to the existing MX beta cycle, which I wouldn't expect going forward
• Checkout the details here. Note the 20% of customers running the stable RC requirement before going GA. This is why you see Dashboard and email prompts to try the new firmware. More customers running RC for the feature enhancements means GA ships sooner.
• I can tell you that we (Meraki) want to move to wired13 as badly as our customers. We also want to ensure our firmware QA process is mature. This round is unusually long because of that.

I'm on your side here but hope that helps provide some color on the situation.

[PhilipDAth]

@Dashboard_DJ I think using a KPI of requiring 20% of your customer base to be using a software image before it is made available as a stable release is not appropriate.  This KPI causes Meraki to promote the use of new beta firmware to its customers - thereby encouraging them to use software not yet considered stable - in production environments just to get to that KPI.

I understand the intent of the KPI - but I think the KPI will cause more grief to customers rather than its goal of producing quality releases.

[Dashboard_DJ]

In fairness, it is considered stable. The name Stable Release Candidate was used for that reason.

[JPAWELCHAK]

Thank you for the prompt and detailed response @Dashboard_DJ

Re. 20% KPI, now I understand why I feel bullied (an admitted exaggeration but pretty close) to instal beta firmware. Truth be known, I've had Support Engineers on multiple occasions demand that we instal beta firmware in order to continue support and testing any further even after asking them to point me to where in the beta release notes where the update addresses the issue we are working to resolve. Our customers and my company can't be what are essentially your beta testers in mission critical environments just so Meraki can achieve a 20% KPI. If I do pull that trigger on a beta firmware upgrade and something does go off the rails how do I explain that to my customers and how do i restore their faith in the Meraki network stack's firmware upgrade process and the services my company provides?

It is my hope this new process will alleviate our dilemma. I also understand why there are some delays switching to the Stable Release Candidate (SRC) model but to have nothing since 12/2016... Shouldn't something have gone out alongside switching to this new model? For clarity what is Meraki's advertised firmware release schedule? Is/was it quarterly as I believe it is/was? Has the schedule changed at some point in the last year? When may we expect to see new releases with the SRC naming nomenclature and process? When may we expect to see a stable release (not SRC) made available? How do we get around Meraki Support Engineers holding our support that we pay dearly for hostage if we do not upgrade to what is/was currently/previously referred to as "beta firmware"?

Thanks again for addressing our concerns and frustrations. We appreciated your time on the matter.

[Dashboard_DJ]

@JPAWELCHAK, I'm not on the Meraki Product team, so I'll my responses here are purely from someone who's also in the field interacting with this stuff on a daily basis. 

There has never been a firmware release schedule, but rather a fairly common 3-4 release cycles per CY. This is clearly well beyond that and they're working to ship this GA as soon as they're able. My understanding is that the last major obstacle to that process is working with a couple of the largest ISPs who use cable modems that don't play with with the new MX static WAN IP assignment in Dashboard. That includes a fail-to-DHCP mechanism which some mainline modems don't respect. 

Hopefully that's useful insight.

[JPAWELCHAK]

@Dashboard_DJ Thanks again, it does indeed. One final clarification on the matter perhaps best addressed from someone on the MX product team or a Meraki Support Manager if available....

How do we get around Meraki Support Engineers holding our support that we pay dearly for hostage if we do not upgrade to what is/was currently/previously referred to as "beta firmware" as we have experienced previously?

[Squuiid]

So, basically the current stable release does not include the following security fixes.
How is this product even considered a viable firewall?!

The practice of not updating your stable branch with security updates for almost a year is ludicrous.

Security fixes:
CVE-2015-3138,CVE-2017-11108,CVE-2017-11541,CVE-2017-11542,CVE-2017-11543,CVE-2017-12893,CVE-2017-12894,CVE-2017-12895,CVE-2017-12896,CVE-2017-12897,CVE-2017-12898,CVE-2017-12899,CVE-2017-12900,CVE-2017-12901,CVE-2017-12902,CVE-2017-12985,CVE-2017-12986,CVE-2017-12987,CVE-2017-12988,CVE-2017-12989,CVE-2017-12990,CVE-2017-12991,CVE-2017-12992,CVE-2017-12993,CVE-2017-12994,CVE-2017-12995,CVE-2017-12996,CVE-2017-12997,CVE-2017-12998,CVE-2017-12999,CVE-2017-13000,CVE-2017-13001,CVE-2017-13002,CVE-2017-13003,CVE-2017-13004,CVE-2017-13005,CVE-2017-13006,CVE-2017-13007,CVE-2017-13008,CVE-2017-13009,CVE-2017-13010,CVE-2017-13011,CVE-2017-13012,CVE-2017-13013,CVE-2017-13014,CVE-2017-13015,CVE-2017-13016,CVE-2017-13017,CVE-2017-13018,CVE-2017-13019,CVE-2017-13020,CVE-2017-13021,CVE-2017-13022,CVE-2017-13023,CVE-2017-13024,CVE-2017-13025,CVE-2017-13026,CVE-2017-13027,CVE-2017-13028,CVE-2017-13029,CVE-2017-13030,CVE-2017-13031,CVE-2017-13032,CVE-2017-13033,CVE-2017-13034,CVE-2017-13035,CVE-2017-13036,CVE-2017-13037,CVE-2017-13038,CVE-2017-13039,CVE-2017-13040,CVE-2017-13041,CVE-2017-13042,CVE-2017-13043,CVE-2017-13044,CVE-2017-13045,CVE-2017-13046,CVE-2017-13047,CVE-2017-13048,CVE-2017-13049,CVE-2017-13050,CVE-2017-13051,CVE-2017-13052,CVE-2017-13053,CVE-2017-13054,CVE-2017-13055,CVE-2017-13687,CVE-2017-13688,CVE-2017-13689,CVE-2017-13690,CVE-2017-13725,CVE-2016-9840,CVE-2016-9841,CVE-2016-9842,CVE-2016-9843,CVE-2016-5131,CVE-2017-3135,CVE-2017-3136,CVE-2017-3137,CVE-2017-3138,CVE-2016-10087,CVE-2016-2217,CVE-2016-8864,CVE-2016-2776,CVE-2016-2775,CVE-2016-8615,CVE-2016-8616,CVE-2016-8617,CVE-2016-8618,CVE-2016-8619,CVE-2016-8620,CVE-2016-8621,CVE-2016-8622,CVE-2016-8623,CVE-2016-8624,CVE-2016-8625,CVE-2016-6304,CVE-2016-6306,CVE-2016-6303,CVE-2016-6302,CVE-2016-2179,CVE-2016-2181,CVE-2016-2182,CVE-2016-2180,CVE-2016-2178,CVE-2016-2177,CVE-2015-8899,CVE-2017-3731,CVE-2017-3732,CVE-2016-7055,CVE-2016-10002,CVE-2016-10003,CVE-2016-9131,CVE-2016-9147,CVE-2016-9444,CVE-2016-7922,CVE-2016-7923,CVE-2016-7924,CVE-2016-7925,CVE-2016-7926,CVE-2016-7927,CVE-2016-7928,CVE-2016-7929,CVE-2016-7930,CVE-2016-7931,CVE-2016-7932,CVE-2016-7933,CVE-2016-7934,CVE-2016-7935,CVE-2016-7936,CVE-2016-7937,CVE-2016-7938,CVE-2016-7939,CVE-2016-7940,CVE-2016-7973,CVE-2016-7974,CVE-2016-7975,CVE-2016-7983,CVE-2016-7984,CVE-2016-7985,CVE-2016-7986,CVE-2016-7992,CVE-2016-7993,CVE-2016-8574,CVE-2016-8575,CVE-2017-5202,CVE-2017-5203,CVE-2017-5204,CVE-2017-5205,CVE-2017-5341,CVE-2017-5342,CVE-2017-5482,CVE-2017-5483,CVE-2017-5484,CVE-2017-5485,CVE-2017-5486
CVE-2016-10229

[JPAWELCHAK]

---

@Squuiid wrote:

So, basically the current stable release does not include the following security fixes.
How is this product even considered a viable firewall?!

The practice of not updating your stable branch with security updates for almost a year is ludicrous.

Security fixes:
CVE-2015-3138,CVE-2017-11108,CVE-2017-11541,CVE-2017-11542,CVE-2017-11543,CVE-2017-12893,CVE-2017-12894,CVE-2017-12895,CVE-2017-12896,CVE-2017-12897,CVE-2017-12898,CVE-2017-12899,CVE-2017-12900,CVE-2017-12901,CVE-2017-12902,CVE-2017-12985,CVE-2017-12986,CVE-2017-12987,CVE-2017-12988,CVE-2017-12989,CVE-2017-12990,CVE-2017-12991,CVE-2017-12992,CVE-2017-12993,CVE-2017-12994,CVE-2017-12995,CVE-2017-12996,CVE-2017-12997,CVE-2017-12998,CVE-2017-12999,CVE-2017-13000,CVE-2017-13001,CVE-2017-13002,CVE-2017-13003,CVE-2017-13004,CVE-2017-13005,CVE-2017-13006,CVE-2017-13007,CVE-2017-13008,CVE-2017-13009,CVE-2017-13010,CVE-2017-13011,CVE-2017-13012,CVE-2017-13013,CVE-2017-13014,CVE-2017-13015,CVE-2017-13016,CVE-2017-13017,CVE-2017-13018,CVE-2017-13019,CVE-2017-13020,CVE-2017-13021,CVE-2017-13022,CVE-2017-13023,CVE-2017-13024,CVE-2017-13025,CVE-2017-13026,CVE-2017-13027,CVE-2017-13028,CVE-2017-13029,CVE-2017-13030,CVE-2017-13031,CVE-2017-13032,CVE-2017-13033,CVE-2017-13034,CVE-2017-13035,CVE-2017-13036,CVE-2017-13037,CVE-2017-13038,CVE-2017-13039,CVE-2017-13040,CVE-2017-13041,CVE-2017-13042,CVE-2017-13043,CVE-2017-13044,CVE-2017-13045,CVE-2017-13046,CVE-2017-13047,CVE-2017-13048,CVE-2017-13049,CVE-2017-13050,CVE-2017-13051,CVE-2017-13052,CVE-2017-13053,CVE-2017-13054,CVE-2017-13055,CVE-2017-13687,CVE-2017-13688,CVE-2017-13689,CVE-2017-13690,CVE-2017-13725,CVE-2016-9840,CVE-2016-9841,CVE-2016-9842,CVE-2016-9843,CVE-2016-5131,CVE-2017-3135,CVE-2017-3136,CVE-2017-3137,CVE-2017-3138,CVE-2016-10087,CVE-2016-2217,CVE-2016-8864,CVE-2016-2776,CVE-2016-2775,CVE-2016-8615,CVE-2016-8616,CVE-2016-8617,CVE-2016-8618,CVE-2016-8619,CVE-2016-8620,CVE-2016-8621,CVE-2016-8622,CVE-2016-8623,CVE-2016-8624,CVE-2016-8625,CVE-2016-6304,CVE-2016-6306,CVE-2016-6303,CVE-2016-6302,CVE-2016-2179,CVE-2016-2181,CVE-2016-2182,CVE-2016-2180,CVE-2016-2178,CVE-2016-2177,CVE-2015-8899,CVE-2017-3731,CVE-2017-3732,CVE-2016-7055,CVE-2016-10002,CVE-2016-10003,CVE-2016-9131,CVE-2016-9147,CVE-2016-9444,CVE-2016-7922,CVE-2016-7923,CVE-2016-7924,CVE-2016-7925,CVE-2016-7926,CVE-2016-7927,CVE-2016-7928,CVE-2016-7929,CVE-2016-7930,CVE-2016-7931,CVE-2016-7932,CVE-2016-7933,CVE-2016-7934,CVE-2016-7935,CVE-2016-7936,CVE-2016-7937,CVE-2016-7938,CVE-2016-7939,CVE-2016-7940,CVE-2016-7973,CVE-2016-7974,CVE-2016-7975,CVE-2016-7983,CVE-2016-7984,CVE-2016-7985,CVE-2016-7986,CVE-2016-7992,CVE-2016-7993,CVE-2016-8574,CVE-2016-8575,CVE-2017-5202,CVE-2017-5203,CVE-2017-5204,CVE-2017-5205,CVE-2017-5341,CVE-2017-5342,CVE-2017-5482,CVE-2017-5483,CVE-2017-5484,CVE-2017-5485,CVE-2017-5486
CVE-2016-10229

---

Agreed and was my point in my original post! Cisco Meraki customers paying a premium for licences that are supposed to include timely firmware updates that simply don't happen unless you are willing to instal what Meraki call "Beta" firmware but then go on to say "Don't worry, its not Beta". Surely you agree Meraki that it can't be both.

Additionally nobody from Meraki is willing to address why its ok for Meraki Support Engineers to withhold support if we do not instal "Beta" firmware in production environments when running into issues as I described previously.

Get on top of this Meraki and get on top of it NOW!

[Dashboard_DJ]

Hi @Squuiid, I definitely hear your concern. Given the delay, the MX team is going to be moving a version of r12 that contains all of the security updates available in r13/14+ to SRC very soon; likely within the next week. 

[ccnewmeraki]

The current v13 build is stable enough and has been out for ages, it fixes a number of critical bugs, people should be encouraged to install it, not put off by calling it a beta.

Given that the a v12 release has gone out with all the security patches from v13/v14; why not make the latest v13 the "stable release candidate", and release the v14 builds to the beta channel?

Anyone currently on a v13 "beta" could then be moved to the "stable release candidate" channel.

If Meraki support is comfortable enough calling the v13 build stable, then the product managers should be acting in a consistent way with that view.

[Uberseehandel]

I'm not sure if it is a Bay Area or a West Coast thing.

But, Meraki is the third network equipment supplier I've used this year (its not over yet) that bangs out "nominally" tested software fixes, under the label "beta" and then expects users to put them into production. One of our major software suppliers from a little further north of the Bay appears to have given up on end-to-end and user-acceptance testing. I'm fed up with it.

Way back when,  we kept a Test Factory that ran two and half shifts per day, seven days per week. This sort of discipline seems to have gone by the board in recent years. I'm told it is all in the name of progress and what the customer wants.

[ccnewmeraki]

The problem is not that there are a number of beta releases, (they would need to make those builds even to just perform traditional in-house testing on them, i'm glad they let us have them), it's the overly cautious approach to the stable releases that is frustrating, and it's quite shocking that it's held up critical security vulnerabilities being patched on an appliance sold as a security appliance!

[DAHG]

---

@ccnewmeraki wrote:

The current v13 build is stable enough and has been out for ages, it fixes a number of critical bugs, people should be encouraged to install it, not put off by calling it a beta.

---

I'm sorry, but that is hogwash. I've been in IT well over 35 years, and over 15 of those of software development. There is a difference between Beta, RC and Stable. There are a lot of industries that are prohibited from running anything but Stable.

If Meraki thinks a firmware version is ready, then move it to the Stable track.

[Uberseehandel]

---

@DAHG wrote:

---

---

I'm sorry, but that is hogwash. I've been in IT well over 35 years, and over 15 of those of software development. There is a difference between Beta, RC and Stable. There are a lot of industries that are prohibited from running anything but Stable.

If Meraki thinks a firmware version is ready, then move it to the Stable track.

---

It isn't just a Meraki thing, it is a West Coast/Industry thing. The continuous drip-feed of software/firmware features which is being adopted by more and more manufacturers, means, in reality, that testing has been moved to the users, and we don't get paid for it. To be honest, Meraki is comparatively good - compared to Juniper, MS, Apple and . . .

I blew my stack at a huge software/hardware co earlier this year and ended up talking to real people just below the  C-level. They admitted that they had just the same problems I was having, like losing access to storage accounts . . .  

In networking, this means that one has to have a test lab, and make one's own mind up about which releases work in the environment into which they will be released into. Not ideal, but better than getting exasperated.

I think if one is running alpha/beta software, one should be given testing scripts and have a formal system to interact with to records results and track faults.

[DAHG]

But we're talking about more than bad/no QA here. We're talking about a company not putting releases to a Stable track, and apparently denying support if you don't load the latest "Beta".

For me if something goes wrong, there is a huge difference in regards to my board and regulators if I can show that the software at fault was Stable as opposed to a 6 month old Beta that the manufacturer said was "stable". It may be the same code, but the implications are wildly different.

I've loaded plenty of RC and Beta releases on non-critical systems. I don't have the luxury of having non-critical firewalls.

[Uberseehandel]

---

@DAHG wrote:

I've loaded plenty of RC and Beta releases on non-critical systems. I don't have the luxury of having non-critical firewalls.

---

Today's reality is that one needs to use test environments that accurately reflect the production environment.

In the next 18 months being able to point at how the manufacturer classifies software releases will no longer serve as a lifesaver in a CYIA defence.

I've always worked with critical systems, what the manufacturer says is immaterial, the onus is on us to get it right.

[CharlesIsWorkin]

The manufacture is supposed to have test environments and rigorous testing methodology. I think it is irresponsible to release beta firmware to the customers at all. All functionality should be tested after each firmware release.

And here's another odd thought, when will the hardware reach it's upgradeable limit? How much firmware can this hardware take? The greater question behind the original question I think is, what are the subscriptions really for?

[DAHG]

True to a certain degree. The manufacturer can't test all environments. Beta is supposed to be ~90% there. Nothing wrong with releasing Beta. Once a Beta release has been out for a while and no major bug reports are coming out, then you release a RC. A RC is supposed to be production ready code. That should attract more customers to load it. Wait a bit, if no bug reports come in, release it as Stable.

[Uberseehandel]

---

@DAHG wrote:

True to a certain degree. The manufacturer can't test all environments. Beta is supposed to be ~90% there. Nothing wrong with releasing Beta. Once a Beta release has been out for a while and no major bug reports are coming out, then you release a RC. A RC is supposed to be production ready code. That should attract more customers to load it. Wait a bit, if no bug reports come in, release it as Stable.

---

Have you spent your entire life drinking the manufacturers' cola?

I can't even begin to express how many ways what you say is wrong. But I guess you are used to it, and have no expectations of anything better. To be honest, software development has gone backwards over the last 40 years.

[DAHG]

---

@Uberseehandel wrote:

---

@DAHG wrote:

True to a certain degree. The manufacturer can't test all environments. Beta is supposed to be ~90% there. Nothing wrong with releasing Beta. Once a Beta release has been out for a while and no major bug reports are coming out, then you release a RC. A RC is supposed to be production ready code. That should attract more customers to load it. Wait a bit, if no bug reports come in, release it as Stable.

---

Have you spent your entire life drinking the manufacturers' cola?

I can't even begin to express how many ways what you say is wrong. But I guess you are used to it, and have no expectations of anything better. To be honest, software development has gone backwards over the last 40 years.

---

No. I have actually spent a good portion of my life as a software developer. Complexity of the deployed environment has gone up. Thus the impossibility for a manufacturer to test every possible scenario. Heck, I spent time doing QA at IBM, we'd get bug reports on LANGUAGE specific version of the software.

I'm not excusing the company. I do however live in the real world. I expect the software to work, I understand that no software beyond a "Hello World" program will have bugs. I expect the manufacturer to do due diligence in testing, hence Beta and RC releases. I also expect that at some point, software becomes RTM.