Cisco Meraki API instability issues

[BasvanH]

Hello,

We have a 24x7 monitoring system which collects data every 5 minutes from the Cisco Meraki API. Since we start using the API we noticed that multiple times a day the API replies with an empty reply.

It happens on multiple calls, this is one of them:

cURL error 52: Empty reply from server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://api.meraki.com/api/v0/networks/N_xxxxxxxxxxx/devices

We are using PHP 8 with Guzzle, but we also had this with PHP 7 and plain CURL commands.

Is anyone else experiencing this? Does Meraki need to fix this on their side? Is there something we can do?

Kind regards,

Bastiaan

[AutomationDude]

Hey Basvan,

We have also experienced this from time to time. To answer your questions, I guess Meraki does need to improve this on their side but since it works more than 99% of the time it's safe to say they are doing a decent job, it's pretty much impossible to guarantee 100%. I don't reckon there's much you can do on your side to fix the issue, but maybe you could set up better error handling within your code for these types of situations so you can retry the call until you get the data you need. Good luck!

[BasvanH]

Ok good to know we're not alone.

Our NOC monitors about 20 API's, polling some every minute, none of them shows as much missed hits than Cisco Meraki does... And closed connection without data is very nasty.

Since we have a centrally placed HTTP client within our code base for all of our remote API's, it's pretty lame to make an exception for failing Meraki's API calls.

Is there some way to reach out to the Meraki tech guys to take a look at this?

[sungod]

In the example you posted it looks like you are using the V0 API which I think is now deprecated, do you see the same with V1?

Fwiw I use the API across several organizations for monitoring/stats/analytics, using quite a few different endpoints, currently with the Python V1 library. Defensive programming with plenty of error recovery is essential, the error recovery in the library is certainly not always enough.

There will be faults at times, some may take quite a while to fix, but there will also be transient faults, including Internet/other issues unrelated to Meraki's infrastructure, all can disrupt things.

One thing that may reduce issues is using an organization's specific shard rather than the generic api.meraki.com, there's another recent thread about this.

You mention polling every minute, if that is on multiple networks/devices in an organization you might also be hitting the rate limit at times.

I guess you prefer the same approach on multiple vendors, but it seems odd to do such frequent polling of an orchestrator that can itself send you alerts as/when they occur, maybe you could get what you need more reliably via webhooks?

[IgorIris]

We are running similar in house developments for multiple organizations' monitoring and have seen lately significant number of issues. Currently it's frequent timeouts on API calls.

The main problem in our experience, that we can not pinpoint any particular patterns when and why we start getting timeouts on some of the calls for some of the organizations.

Regarding 429 error code returned by Meraki API -  we worked out a solution that allows us to limit the calls number to conform with Meraki rate limiting. Yet, from time to time we still starting to get lots of 429 errors, one of the solutions we found is to change the egress IP we are running from, but it's work round that causes us a lot time spent on these. Has  anyone seen similar problems?

[webfrank]

Hi, in the latest days we are experiencing the same problem, it seems that instead of rate limiting per organization they are limiting per source ip.

We switched to shard address but it's not the right solution, just a workaround.

I have on opened case but no news so far.

[BasvanH]

@webfrank keep us posted regarding your case.

We respect the rate limits and 429's, but the issue is just random. Out of the blue some calls are just not handled properly.

We are not polling all API's every minute, but some, thats was just to show that the error rate of the Meraki API is way higher than API's we call more frequently.. We have our reasons to do so, Meraki API we hit every 5 minutes, and just two organizations on n146, and I see more post about n146 being slow.

[webfrank]

Hi @BasvanH , the issue seems solved (not fully tested) but the case is still open and waiting for an official reply.

[BasvanH]

Hi @webfrank, I notice a reduced amount of failed requests. But I do still have them occasionally, as in once or twice a day. 

[webfrank]

Hi, Meraki closed the ticket. At this time we are not experiencing issues. Are you?