Meraki API regular maintenance

[winadm1]

Good day!

We use Meraki API to perform some kind of 24/7 monitoring, and on a regular basis we experience API unavailability in night hours, here are some statistics in UTC +3 TZ

05/10 02:01
28/09 03:19
20/09 02:11
14/09 23:14
08/09 02:57
09/08 02:46
03/08 00:55

Usually, API gets back in 20-30 minutes.

And we don't get any notifications about planned API maintenance. So the question is if anyone else got this problem too? As API selfheals in 20-30 minutes the problem is absolutely uncritical, but we are a little worried every time 🙂

[sungod]

I have scripts that run every few minutes on some organizations, there's no sign of similar behaviour.

Are these outages you see on a particular shard? If it is then I'd open a support case.

[winadm1]

Our exporter every minute queries https://api.meraki.com/api/v1/organizations endpoint for org list and then goes deeper to get some metrics.

And on the first query, we got nginx 502 Bad Gateway.

So all deeper queries predictably failed with the same 502 status. And then some time after API selfheals and everything works as expected.

Even today's night UTC +3 TZ we got 502 for 5 minutes

from 2023-10-07 05:41:41

to 2023-10-07 05:47:37

[PhilipDAth]

>And on the first query, we got nginx 502 Bad Gateway.

I think that happens when you significantly exceed the rate limit.  Any chance you are doing a high rate of calls per second and ignoring the 429 error?

[winadm1]

Never seen 429 on this exporter, and we are logging all the responses. All calls are done sequentially, no simultaneous/async calls, so rate limit is hardly a problem from the first sight.

As a test we can move exporter to another cloud to check, if current infa is somehow a reason for such API behavior. But as nobody is aware of the problem - I think we will just mute alerts about API unavailability at night hours.

[John-K]

Please feel free to open a support case on this. What you've described does not sound like expected behavior.