HTTP 500 and 502 Error (Page not found) while using API on a very specific Org
Our API to one specific org started to receive HTTP Status code 500 and 502 Error from the response header. This is not constantly happening and non-predictable. It may work fine for a few days and suddenly we may have 3 or 5 "Page not found" during a short period like less than 60 mins.
All these 500 and 502 response body were the same "Page not Found". (I will paste the full response body)
What we have identified were :
This issue is only on one specific Org
Happens only to "POST" requests, not on "GET"
Known endpoints with these issues are: "createOrganizationNetwork" "bindNetwork" "createOrganizationActionBatch" "removeNetworkDevice"
All response body was the same
The whole API process duration from send to receive varies from a few seconds up to 61 seconds
from 1st May to 4 June we had 25 recorded 500 Errors and 15 recorded 502 Errors
Anyone has similar issues or experienced something similar?
Contacted Meraki TAC, came back with
"didn't find any issue on our end. No backend errors were generated on the shard when the API request failed which means the affected requests never made to the shard. The load on the shard was normal and was working as normal during the whole time. No outages were reported matching the provided timestamps and the service was available throughout the whole time."
"The error 502 and 500 are related to the unavailability of web service. This can be related to end device/network issue or the service is not available. In our case, we didn't see any error getting registered on the shard which are usually there if it is a service issue."
"The error can also be caused when there is a delay between the client and the server due to packet loss or the request does not make in the time allowed. The issue is very much looking at the client-side. When I mentioned 'unavailability of web service' it was in the context of the API call not reaching to the server."
This is extremely fascinating to us, Meraki API is 100% using TCP and should be connection-oriented. Between our end to the API endpoints, there will be NO packet loss as we successfully received the 500/502 with a full response body.
I think there must be some performance issue on that Org hosted environment, however, as a customer, we don't have any visibility of that and have to trust support to help us to identify the root cause.
Re: HTTP 500 and 502 Error (Page not found) while using API on a very specific Org
Back in April I had a similar problem in development/test of a daily scheduled API extract that ran just after 0000 UTC, there would be short burst of 'page not found' errors, on c. 10-20% of the runs, affecting just a few calls out of many in a run.
In case around 0000 UTC is a busy or somehow 'bad' time, I thought I'd shift things a bit later, there were no further errors.
However, this was during development/test, and there were other changes in the next few days, in particular:
- using api.meraki.com instead of api-mp.meraki.com
- moved over to the API V1 beta
...it might be worth experimenting with these.
Since late April there've been no further errors, even though the extracts are running against multiple orgs now.