Our API to one specific org started to receive HTTP Status code 500 and 502 Error from the response header. This is not constantly happening and non-predictable. It may work fine for a few days and suddenly we may have 3 or 5 "Page not found" during a short period like less than 60 mins.
All these 500 and 502 response body were the same "Page not Found". (I will paste the full response body)
What we have identified were :
- This issue is only on one specific Org
- Happens only to "POST" requests, not on "GET"
- Known endpoints with these issues are: "createOrganizationNetwork" "bindNetwork" "createOrganizationActionBatch" "removeNetworkDevice"
- All response body was the same
- The whole API process duration from send to receive varies from a few seconds up to 61 seconds
- from 1st May to 4 June we had 25 recorded 500 Errors and 15 recorded 502 Errors
Anyone has similar issues or experienced something similar?
Contacted Meraki TAC, came back with
"didn't find any issue on our end. No backend errors were generated on the shard when the API request failed which means the affected requests never made to the shard. The load on the shard was normal and was working as normal during the whole time. No outages were reported matching the provided timestamps and the service was available throughout the whole time."
"The error 502 and 500 are related to the unavailability of web service. This can be related to end device/network issue or the service is not available. In our case, we didn't see any error getting registered on the shard which are usually there if it is a service issue."
"The error can also be caused when there is a delay between the client and the server due to packet loss or the request does not make in the time allowed. The issue is very much looking at the client-side. When I mentioned 'unavailability of web service' it was in the context of the API call not reaching to the server."
This is extremely fascinating to us, Meraki API is 100% using TCP and should be connection-oriented. Between our end to the API endpoints, there will be NO packet loss as we successfully received the 500/502 with a full response body.
I think there must be some performance issue on that Org hosted environment, however, as a customer, we don't have any visibility of that and have to trust support to help us to identify the root cause.
Response Boby:
"<!DOCTYPE html PUBLIC \\\"-\\/\\/W3C\\/\\/DTD XHTML 1.0 Transitional\\/\\/EN\\\" \\\"http:\\/\\/www.w3.org\\/TR\\/xhtml1\\/DTD\\/xhtml1-transitional.dtd\\\">\\n<html xmlns=\\\"http:\\/\\/www.w3.org\\/1999\\/xhtml\\\" xml:lang=\\\"en\\\" lang=\\\"en\\\">\\n<head>\\n <title>Page not found - Cisco Meraki<\\/title>\\n <style type=\\\"text\\/css\\\">\\n body {\\n display: flex;\\n flex-direction: column;\\n justify-content: flex-start;\\n align-items: center;\\n margin: 0 2em;\\n font-family: Helvetica, Arial, sans-serif;\\n line-height: 1.45;\\n color: #222325; \\/* same as $gray-10 in mkiColorVariables.json *\\/\\n }\\n .header__container,\\n .content__container,\\n .footer__container {\\n max-width: 800px;\\n width: 100%;\\n }\\n .header__container {\\n padding: 2em 0;\\n border-bottom: 1px solid #D7D7D9; \\/* same as $gray-80 in mkiColorVariables.json *\\/\\n }\\n .header__merakiLogo {\\n height: 2.125em;\\n }\\n .header__merakiLogo--gray {\\n fill: #898b8e;\\n }\\n .header__merakiLogo--green {\\n fill: #67b346;\\n }\\n h1.content__title {\\n font-size: 1.728em; \\/* from typographyBase.scss *\\/\\n font-weight: 300;\\n margin-bottom: 1em;\\n }\\n .content__container {\\n margin-bottom: 1em;\\n }\\n .content__container a {\\n color: #1D770B; \\/* same as $linkColor in variables.scss *\\/\\n }\\n .footer__container {\\n padding: 1em 0;\\n text-align: right;\\n border-top: 1px solid #D7D7D9; \\/* same as $gray-80 in mkiColorVariables.json *\\/\\n }\\n .footer__text {\\n font-size: 0.833em;\\n color: #898A8C; \\/* same as $gray-50 in mkiColorVariables.json *\\/\\n }\\n <\\/style>\\n<\\/head>\\n<body>\\n <div class=\\\"header__container\\\">\\n <svg class=\\\"header__merakiLogo\\\" xmlns=\\\"http:\\/\\/www.w3.org\\/2000\\/svg\\\" viewBox=\\\"0 0 130 25\\\"><title>Cisco Meraki<\\/title><rect class=\\\"header__merakiLogo--gray\\\" x=\\\"13.29\\\" y=\\\"16.46\\\" width=\\\"2.06\\\" height=\\\"8.17\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M32,18.8a3.64,3.64,0,0,0-1.73-.44,2.18,2.18,0,1,0,0,4.36A3.63,3.63,0,0,0,32,22.29v2.19a6.53,6.53,0,0,1-1.88.29,4.23,4.23,0,1,1,0-8.46,6.29,6.29,0,0,1,1.88.29Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M10.46,18.8a3.59,3.59,0,0,0-1.73-.44,2.18,2.18,0,1,0,0,4.36,3.59,3.59,0,0,0,1.73-.43v2.19a6.49,6.49,0,0,1-1.88.29,4.23,4.23,0,1,1,0-8.46,6.23,6.23,0,0,1,1.88.29Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M38.45,18.4a2.15,2.15,0,1,0,2.13,2.15,2.11,2.11,0,0,0-2.13-2.15m4.31,2.15a4.31,4.31,0,1,1-4.31-4.23,4.2,4.2,0,0,1,4.31,4.23\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M23.17,18.32a7.05,7.05,0,0,0-1.61-.25c-.83,0-1.27.27-1.27.67s.6.67.94.78l.57.18A2.46,2.46,0,0,1,23.74,22c0,2.05-1.81,2.74-3.38,2.74a12.56,12.56,0,0,1-2.22-.22V22.67a7.73,7.73,0,0,0,1.95.31c1,0,1.5-.3,1.5-.77s-.41-.65-.92-.81l-.44-.14c-1.15-.36-2.1-1-2.1-2.39,0-1.53,1.14-2.55,3-2.55a8.91,8.91,0,0,1,2,.26Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M2,7.68a1,1,0,0,0-2,0V9.83a1,1,0,1,0,2,0Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M7.66,4.87a1,1,0,0,0-2,0v5a1,1,0,0,0,2,0Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M13.28,1a1,1,0,0,0-2,0V11.87a1,1,0,1,0,2,0Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M18.9,4.87a1,1,0,0,0-2,0v5a1,1,0,1,0,2,0Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M24.52,7.68a1,1,0,0,0-2,0V9.83a1,1,0,1,0,2,0Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M30.14,4.87a1,1,0,0,0-2,0v5a1,1,0,1,0,2,0Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M35.77,1a1,1,0,1,0-2,0V11.87a1,1,0,1,0,2,0Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M41.39,4.87a1,1,0,0,0-2.06,0v5a1,1,0,0,0,2.06,0Z\\\"\\/><path class=\\\"header__merakiLogo--gray\\\" d=\\\"M47,7.68a1,1,0,0,0-2,0V9.83a1,1,0,1,0,2,0Z\\\"\\/><path class=\\\"header__merakiLogo--green\\\" d=\\\"M68.43,24.64,62.06,8.9V24.64H60.73V7.13h2l6.44,16,6.08-16h1.95V24.64H75.9V9l-6,15.61Z\\\"\\/><path class=\\\"header__merakiLogo--green\\\" d=\\\"M91.28,21.18A5.11,5.11,0,0,1,86,25c-3.43,0-6-2.36-6-6.69s2.5-6.69,5.76-6.69,5.71,2.19,5.71,7H81.36c.12,3.6,2.16,5.25,4.69,5.25a3.77,3.77,0,0,0,3.84-2.7Zm-9.9-3.65h8.75c-.17-3.24-2-4.84-4.35-4.84S81.63,14.37,81.38,17.54Z\\\"\\/><path class=\\\"header__merakiLogo--green\\\" d=\\\"M94.57,24.64V12h1.07l.19,2.43a3.67,3.67,0,0,1,3.55-2.8,4.51,4.51,0,0,1,1.31.2v1.29a4.93,4.93,0,0,0-1.39-.22c-1.85,0-3.43,1.8-3.43,4.84v6.91Z\\\"\\/><path class=\\\"header__merakiLogo--green\\\" d=\\\"M102.35,15.28c.19-2.16,2.12-3.65,4.91-3.65S112,13.21,112,15.79v8.85h-1.07l-.19-2.09A5,5,0,0,1,106.22,25c-2.46,0-4.23-1.33-4.23-3.57s1.31-3.48,4.86-4.06l3.79-.63v-1.1c0-1.82-1.29-2.94-3.45-2.94s-3.31,1-3.48,2.58Zm8.29,4.86V17.78l-3.4.58c-3,.51-3.89,1.46-3.89,3,0,1.7,1.31,2.55,3.21,2.55A4,4,0,0,0,110.64,20.14Z\\\"\\/><path class=\\\"header__merakiLogo--green\\\" d=\\\"M115.32,24.64V7.13h1.31v10L122.69,12h1.78L118,17.51l7.3,7.12h-1.77L116.63,18v6.69Z\\\"\\/><path class=\\\"header__merakiLogo--green\\\" d=\\\"M126.92,8.93V7.13h1.58v1.8ZM127,24.64V12h1.31V24.64Z\\\"\\/><\\/svg>\\n <\\/div>\\n <div class=\\\"content__container\\\">\\n <h1 class=\\\"content__title\\\">Page not found<\\/h1>\\n <p class=\\\"content__primaryText\\\">The page you are looking for may have been moved or does not exist.<\\/p>\\n <p class=\\\"content__secondaryText\\\">To log in to the Cisco Meraki Dashboard, go to <a href=\\\"https:\\/\\/dashboard.meraki.com\\\">https:\\/\\/dashboard.meraki.com<\\/a>.<\\/p>\\n <\\/div>\\n <div class=\\\"footer__container\\\">\\n <span id=\\\"footer\\\" class=\\\"footer__text\\\">© Cisco Systems, Inc.<\\/span>\\n <\\/div>\\n <script type=\\\"text\\/javascript\\\" charset=\\\"utf-8\\\">\\n \\/\\/<![CDATA[\\n (function () {\\n var year = new Date().getFullYear();\\n document.getElementById(\\\"footer\\\").innerHTML = '© ' + year + ' Cisco Systems, Inc.';\\n })()\\n \\/\\/]]>\\n <\\/script>\\n<\\/body>\\n<\\/html>"
Converted the HTML response to this image :
