HTTP 500 and 502 Error (Page not found) while using API on a very specific Org

Here to help

HTTP 500 and 502 Error (Page not found) while using API on a very specific Org

Our API to one specific org started to receive HTTP Status code 500 and 502 Error from the response header. This is not constantly happening and non-predictable. It may work fine for a few days and suddenly we may have 3 or 5 "Page not found" during a short period like less than 60 mins.


All these 500 and 502 response body were the same "Page not Found". (I will paste the full response body)


What we have identified were :

  1. This issue is only on one specific Org
  2. Happens only to "POST" requests, not on "GET"
  3. Known endpoints with these issues are: "createOrganizationNetwork" "bindNetwork" "createOrganizationActionBatch" "removeNetworkDevice"
  4. All response body was the same
  5. The whole API process duration from send to receive varies from a few seconds up to 61 seconds
  6. from 1st May to 4 June we had 25 recorded 500 Errors and 15 recorded 502 Errors


Anyone has similar issues or experienced something similar?


Contacted Meraki TAC, came back with


"didn't find any issue on our end. No backend errors were generated on the shard when the API request failed which means the affected requests never made to the shard. The load on the shard was normal and was working as normal during the whole time. No outages were reported matching the provided timestamps and the service was available throughout the whole time."


"The error 502 and 500 are related to the unavailability of web service. This can be related to end device/network issue or the service is not available. In our case, we didn't see any error getting registered on the shard which are usually there if it is a service issue."


"The error can also be caused when there is a delay between the client and the server due to packet loss or the request does not make in the time allowed. The issue is very much looking at the client-side. When I mentioned 'unavailability of web service' it was in the context of the API call not reaching to the server."


This is extremely fascinating to us, Meraki API is 100% using TCP and should be connection-oriented. Between our end to the API endpoints, there will be NO packet loss as we successfully received the 500/502 with a full response body.


I think there must be some performance issue on that Org hosted environment, however, as a customer, we don't have any visibility of that and have to trust support to help us to identify the root cause.


Response Boby: 

"<!DOCTYPE html PUBLIC \\\"-\\/\\/W3C\\/\\/DTD XHTML 1.0 Transitional\\/\\/EN\\\" \\\"http:\\/\\/\\/TR\\/xhtml1\\/DTD\\/xhtml1-transitional.dtd\\\">\\n<html xmlns=\\\"http:\\/\\/\\/1999\\/xhtml\\\" xml:lang=\\\"en\\\" lang=\\\"en\\\">\\n<head>\\n <title>Page not found - Cisco Meraki<\\/title>\\n <style type=\\\"text\\/css\\\">\\n body {\\n display: flex;\\n flex-direction: column;\\n justify-content: flex-start;\\n align-items: center;\\n margin: 0 2em;\\n font-family: Helvetica, Arial, sans-serif;\\n line-height: 1.45;\\n color: #222325; \\/* same as $gray-10 in mkiColorVariables.json *\\/\\n }\\n .header__container,\\n .content__container,\\n .footer__container {\\n max-width: 800px;\\n width: 100%;\\n }\\n .header__container {\\n padding: 2em 0;\\n border-bottom: 1px solid #D7D7D9; \\/* same as $gray-80 in mkiColorVariables.json *\\/\\n }\\n .header__merakiLogo {\\n height: 2.125em;\\n }\\n .header__merakiLogo--gray {\\n fill: #898b8e;\\n }\\n .header__merakiLogo--green {\\n fill: #67b346;\\n }\\n h1.content__title {\\n font-size: 1.728em; \\/* from typographyBase.scss *\\/\\n font-weight: 300;\\n margin-bottom: 1em;\\n }\\n .content__container {\\n margin-bottom: 1em;\\n }\\n .content__container a {\\n color: #1D770B; \\/* same as $linkColor in variables.scss *\\/\\n }\\n .footer__container {\\n padding: 1em 0;\\n text-align: right;\\n border-top: 1px solid #D7D7D9; \\/* same as $gray-80 in mkiColorVariables.json *\\/\\n }\\n .footer__text {\\n font-size: 0.833em;\\n color: #898A8C; \\/* same as $gray-50 in mkiColorVariables.json *\\/\\n }\\n <\\/style>\\n<\\/head>\\n<body>\\n <div class=\\\"header__container\\\">\\n <svg class=\\\"header__merakiLogo\\\" xmlns=\\\"http:\\/\\/\\/2000\\/svg\\\" viewBox=\\\"0 0 130 25\\\"><title>Cisco Meraki<\\/title><rect class=\\\"header__merakiLogo--gray\\\" x=\\\"13.29\\\" y=\\\"16.46\\\" width=\\\"2.06\\\" height=\\\"8.17\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M32,18.8a3.64,3.64,0,0,0-1.73-.44,2.18,2.18,0,1,0,0,4.36A3.63,3.63,0,0,0,32,22.29v2.19a6.53,6.53,0,0,1-1.88.29,4.23,4.23,0,1,1,0-8.46,6.29,6.29,0,0,1,1.88.29Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M10.46,18.8a3.59,3.59,0,0,0-1.73-.44,2.18,2.18,0,1,0,0,4.36,3.59,3.59,0,0,0,1.73-.43v2.19a6.49,6.49,0,0,1-1.88.29,4.23,4.23,0,1,1,0-8.46,6.23,6.23,0,0,1,1.88.29Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M38.45,18.4a2.15,2.15,0,1,0,2.13,2.15,2.11,2.11,0,0,0-2.13-2.15m4.31,2.15a4.31,4.31,0,1,1-4.31-4.23,4.2,4.2,0,0,1,4.31,4.23\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M23.17,18.32a7.05,7.05,0,0,0-1.61-.25c-.83,0-1.27.27-1.27.67s.,2.46,0,0,1,23.74,22c0,2.05-1.81,2.74-3.38,2.74a12.56,12.56,0,0,1-2.22-.22V22.67a7.73,7.73,0,0,0,1.95.31c1,0,1.5-.3,1.5-.77s-.41-.65-.92-.81l-.44-.14c-1.15-.36-2.1-1-2.1-2.39,0-1.53,1.14-2.55,3-2.55a8.91,8.91,0,0,1,2,.26Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M2,7.68a1,1,0,0,0-2,0V9.83a1,1,0,1,0,2,0Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M7.66,4.87a1,1,0,0,0-2,0v5a1,1,0,0,0,2,0Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M13.28,1a1,1,0,0,0-2,0V11.87a1,1,0,1,0,2,0Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M18.9,4.87a1,1,0,0,0-2,0v5a1,1,0,1,0,2,0Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M24.52,7.68a1,1,0,0,0-2,0V9.83a1,1,0,1,0,2,0Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M30.14,4.87a1,1,0,0,0-2,0v5a1,1,0,1,0,2,0Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M35.77,1a1,1,0,1,0-2,0V11.87a1,1,0,1,0,2,0Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M41.39,4.87a1,1,0,0,0-2.06,0v5a1,1,0,0,0,2.06,0Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M47,7.68a1,1,0,0,0-2,0V9.83a1,1,0,1,0,2,0Z\\\"\\/><path class=\\\"header__merakiLogo--green\\\" d=\\\"M68.43,24.64,62.06,8.9V24.64H60.73V7.13h2l6.44,16,6.08-16h1.95V24.64H75.9V9l-6,15.61Z\\\"\\/><path class=\\\"header__merakiLogo--green\\\" d=\\\"M91.28,21.18A5.11,5.11,0,0,1,86,25c-3.43,0-6-2.36-6-6.69s2.5-6.69,5.76-6.69,5.71,2.19,5.71,7H81.36c.12,3.6,2.16,5.25,4.69,5.25a3.77,3.77,0,0,0,3.84-2.7Zm-9.9-3.65h8.75c-.17-3.24-2-4.84-4.35-4.84S81.63,14.37,81.38,17.54Z\\\"\\/><path class=\\\"header__merakiLogo--green\\\" d=\\\"M94.57,24.64V12h1.07l.19,2.43a3.67,3.67,0,0,1,3.55-2.8,4.51,4.51,0,0,1,1.31.2v1.29a4.93,4.93,0,0,0-1.39-.22c-1.85,0-3.43,1.8-3.43,4.84v6.91Z\\\"\\/><path class=\\\"header__merakiLogo--green\\\" d=\\\"M102.35,15.28c.19-2.16,2.12-3.65,4.91-3.65S112,13.21,112,15.79v8.85h-1.07l-.19-2.09A5,5,0,0,1,106.22,25c-2.46,0-4.23-1.33-4.23-3.57s1.31-3.48,4.86-4.06l3.79-.63v-1.1c0-1.82-1.29-2.94-3.45-2.94s-3.31,1-3.48,2.58Zm8.29,4.86V17.78l-3.4.58c-3,.51-3.89,1.46-3.89,3,0,1.7,1.31,2.55,3.21,2.55A4,4,0,0,0,110.64,20.14Z\\\"\\/><path class=\\\"header__merakiLogo--green\\\" d=\\\"M115.32,24.64V7.13h1.31v10L122.69,12h1.78L118,17.51l7.3,7.12h-1.77L116.63,18v6.69Z\\\"\\/><path class=\\\"header__merakiLogo--green\\\" d=\\\"M126.92,8.93V7.13h1.58v1.8ZM127,24.64V12h1.31V24.64Z\\\"\\/><\\/svg>\\n <\\/div>\\n <div class=\\\"content__container\\\">\\n <h1 class=\\\"content__title\\\">Page not found<\\/h1>\\n <p class=\\\"content__primaryText\\\">The page you are looking for may have been moved or does not exist.<\\/p>\\n <p class=\\\"content__secondaryText\\\">To log in to the Cisco Meraki Dashboard, go to <a href=\\\"https:\\/\\/\\\">https:\\/\\/<\\/a>.<\\/p>\\n <\\/div>\\n <div class=\\\"footer__container\\\">\\n <span id=\\\"footer\\\" class=\\\"footer__text\\\">&copy; Cisco Systems, Inc.<\\/span>\\n <\\/div>\\n <script type=\\\"text\\/javascript\\\" charset=\\\"utf-8\\\">\\n \\/\\/<![CDATA[\\n (function () {\\n var year = new Date().getFullYear();\\n document.getElementById(\\\"footer\\\").innerHTML = '&copy; ' + year + ' Cisco Systems, Inc.';\\n })()\\n \\/\\/]]>\\n <\\/script>\\n<\\/body>\\n<\\/html>"


Converted the HTML response to this image : 



Head in the Cloud

Back in April I had a similar problem in development/test of a daily scheduled API extract that ran just after 0000 UTC, there would be short burst of 'page not found' errors, on c. 10-20% of the runs, affecting just a few calls out of many in a run.


In case around 0000 UTC is a busy or somehow 'bad' time, I thought I'd shift things a bit later, there were no further errors.


However, this was during development/test, and there were other changes in the next few days, in particular:

- using instead of

- moved over to the API V1 beta might be worth experimenting with these.


Since late April there've been no further errors, even though the extracts are running against multiple orgs now.

Thank you @sungod, that's interesting as you said after 0000 UTC. Maybe I should have some constant runs to see if similar in that Org.

We may also evaluate API V1 beta.
Get notified when there are additional replies to this discussion.