Unable to fetch configuration - Unresolved software bug

[General-Zod]

Greetings,

I know this has been flagged previously in other threads however I thought I'd put a bit more focus on it.

We have a customer (large retailer) who has a fleet of MX64's. In the last 3 weeks we have had 5 occurances where "Unable to fetch configuration" was displayed on the dashboard. This co-incides with the MX going offline - at least from the dashboard's perspective.

Customer was running 14.39 software. Whilst a call to Meraki resolved the issue, they had to fix something in the backend concerning local storage on the MX, the customer is still impacted. Apparently Meraki fixed and flagged this behaviour as a rare occurance according to release notes, 14.35, thanks @PhilipDAth 

"Resolved an issue that could result in devices failing to pull configuration updates from the Meraki Dashboard in rare circumstances"

So not only is the issue _not_ fixed, it certainly is _not_ a rare circumstance. Too boot the Meraki employee I spoke to informed me that this issue is fixed in 14.40 despite the rls notes not stating such. Was informed by same individual not all bug fixes are stated in the release notes. I took it on face value and requested the customer to upgrade to 14.40 to prevent the issue re-occuring, which they did only to find that another event of the issue occured.

Was informed by different Meraki support engineer that this bug is still present and it's still under investigation. 

This does not sit well with my customer, especially coming into the Christmas period.

Have requested if Meraki can proactivily perform some action to prevent downtime during business hours but to no avail. Have also requested list of devices in which the storage space is critical in a effort to help identify which units will suffer the same fate, no avail.

Have also been informed that even the RC and BETA software don't have any fix for this issue.

These units were installed approximately 3 years as part of a large project and it's almost as if the internal storage has been filling up eversince and reached a critial threshold causing this anomoly, so I suspect that most units that are part of this fleet will have the same fate at some stage.

There is a detailed internal document regarding this issue however it's not public facing.

Have asked Meraki to reference the bug ID and a commitment that the issue has high visibility with L4 engineering. Hopefully my customer will swallow that instead of a RCA.

Cheers

[PhilipDAth]

That is yucky.

>Have requested if Meraki can proactivily perform some action to prevent downtime during business hours

About the only thing you could do yourself if you think the issue is related to storage space on the device is a pro-active factory reset.  Not very nice.

[General-Zod]

...very yucky.

They have dual WAN configured also, so would need tech onsite to change LAN4 > WAN2, ip details etc.

definately not ideal.

[General-Zod]

Just a update for those of you who are also experiencing the problem.

Problem is due to internal storage not being mounted correctly.

Meraki stated that v14.50 (Stable release Candidate) fixes this problem - as per release notes.

Unfortunately the problem of Unable to fetch configuration still remains. 

Meraki support informed me that the identified bug has been fixed in version 14.50 and this new "unknown" bug displays the same symptoms resulting in the storage being dismounted and ultimately displaying "Unable to fetch configuration" - this is of no comfort to the customer.

Seen 3 occurances of this since moving to rls 14.50 in as many weeks. 

Have seen over 10 occurances of this issue on 14.49 since Nov 2019 at difference sites

This has been esculated to Meraki and they have assured me this is a high priority and they are working on a "real" fix.

At any moment in time a site can go off the air. This is supposed to be a rare occurance despite it happening at 13 sites, same customer.

Customer not happy.

[CountRushmore]

This issue continue to exist for us as well. We have had multiple sites out of a 50+ site network experience this including one today on the latest 14.53 release.

Meraki support and sales contacts have been unable to provide any substantive update which indicates they understand the issue or are working towards a resolution.

[General-Zod]

That's a shame @CountRushmore 

A big thank you for being the guinea pig though, you have saved me a lot of time and effort.

We are still running on 14.50, since then we have experienced 15 x "Unable to fetch" occurances. Approx 3 per month.

No end in sight, despite esculating this with Meraki. 👿

Has been dragging on since Q4 - 2019 

[CountRushmore]

Our account team will not even return our emails anymore so I'll probably be requesting a new account team in the next few days.

Anyone have any experience with that?

[CameronMoody]

Hi Everyone,

Edit/Correction: @Bsalami below has a reply to this particular issue, which is related to 'unable to fetch config' issues occurring before July 2020. 

The 64/65 hardware replacement that I referenced has similar symptoms with an 'unable to fetch config' warning, but is a different issue that applies to devices running very old firmware versions, and affected customers have been contacted. If you're experiencing an 'unable to fetch config' issue on MX64/65 devices and have a dashboard notification after July 2020, you should refer to that document.

If you have any questions or concerns, I recommend reaching out to Cisco Meraki Support and referring to the documentation if you think you may be affected by this issue.

[CountRushmore]

@CameronMoody are there overlapping issues here? We have been experiencing this since November/December 2019 so a certificate expiring in July 2020 doesn't exactly line up.

[Bsalami]

The majority of issues related to "unable to fetch config" affecting the MX64/65 models have been resolved on the latest beta version MX 15.33 available on Dashboard.

[DarrenOC]

Hi @CameronMoody 

Will there be a proactive replacement program like what happened with the noisey fan issue a couple of years ago?