Error: 429 Client Error: Too Many Requests

[Adrian4]

Hello,

Recently I have been running into this unexpectantly. If I wait 5 or 10 mins it goes away but it kept happening so I ran the API requests summary endpoint each time it happened and I saw thousands of requests with a 429 response.

Since 429 is the too many requests error, I assumed a first call 429 and then it re-try's a thousands times immediately after until it gives up which is what gives me all those 429?


I tried to dig deeper into where they were coming from....

all the 429 requests were apparently getNetworkWirelessClientCountHistory ?!? - that doesn't make any sense as i only ran that once and it was successful.

also - in the thousands, they seem to alternate from ssid 4 to ssid14 over and over - not sure what that is about as I am not jumping around SSIDs and the one im connected to isnt 4 or 14.


can anyone shed some light?

[sungod]

Assume you are using the Meraki Python library.

Are you telling the library to wait on retry? This is important to avoid retries being made too fast.

Combined with setting a generous retry limit should ensure calls eventually complete ok.

        wait_on_rate_limit=True,
        maximum_retries=100

For instance...

    async with meraki.aio.AsyncDashboardAPI(
        api_key=API_KEY,
        base_url='https://api.meraki.com/api/v1/',
        print_console=False,
        output_log=False,
        suppress_logging=True,
        wait_on_rate_limit=True,
        maximum_retries=100
    ) as aiomeraki:

If you are using some different code to handle 429s, make sure it is obeying the Retry-After header in the 429 response, see https://developer.cisco.com/meraki/api-v1/rate-limit/

[Adrian4]

thank you! to be honest I had never come across 429 errors until a day or two ago 😛

[PhilipDAth]

I haven't checked - but are you saying wait_on_rate_limit is not True by default?  I'm gobsmacked.

[sungod]

Tbh I think it defaults to on, but decades of coding paranoia leads me to always set it explicitly in my scripts 😀

[PhilipDAth]

I checked, and wait_on_rate_limit is on by default.

https://github.com/meraki/dashboard-api-python/blob/main/meraki/config.py#L19

and maximum_retries defaults to 2.

https://github.com/meraki/dashboard-api-python/blob/main/meraki/config.py#L37 

And for anyone finding this using Google, here are all the Meraki Python library defaults:

https://github.com/meraki/dashboard-api-python/blob/main/meraki/config.py 

[Adrian4]

the problem has returned 😞

every so often getting 429 errors and when I check the summary I see this  "429": 1307,

So two things - one, when it gets a 429, it is clearly re-trying over thousand times in tiny time frame - 
not sure I am using these variable properly

wait_on_rate_limit=True,
maximum_retries=100

Am I supposed to pass them as part of the API headers or something?

Secondly, I must still be doing something to trigger the 429 rate limit in the first place. The limit is 5 requests per second I think? After each and every request call I have put in a 0.3 second delay which should mean I cant be sending more than 4 a second so how is this happening?

To produce the delay I am using the time module and setting a global variable of

DELAY = 0.3

then after every request putting the line 

time.sleep(DELAY)

networks = requests.get(networks_url, headers=headers, verify=f"{root_path}/Cisco Umbrella Root CA.crt")
time.sleep(DELAY)

[sungod]

Are you using the Meraki Python library?

[Adrian4]

Im using "requests" HTTP library

[sungod]

Then you are not using the Meraki Python library.

These set behaviour of the library, if you are not using it then they will not help you.

wait_on_rate_limit=True,
maximum_retries=100

I recommend using the library.

If don't want to use it, you will need to do as suggested above and make sure you code to obey the Retry-After header in the 429 response, see https://developer.cisco.com/meraki/api-v1/rate-limit/

For instance based on the example on the linked page...

response = requests.request("GET", url, headers=headers)
​
if response.status_code == 200:
    # Success logic
elif response.status_code == 429:
    time.sleep(int(response.headers["Retry-After"]))
else:
    # Handle other response codes

[Adrian4]

Hi,

Thanks for the reply! makes sense.

However, the 429 error means iv already hit the rate limit, so adding a sleep timer on that code will help reduce the amount of time i have to wait before i can send again (by preventing the re-try spam) but I still have the issue of hitting the limit in the first place 😞

[sungod]

The rate limit is just a fact of life, but if you use the back-off and retry mechanism your calls will eventually succeed.

For example, I have scripts using the Meraki Python library aio functions that could potentially make thousands*** of calls 'at once', but the rate limit and retry handling kicks in, things back off and retry, resulting in throughput at the rate limit, the library handles it all for me.

The alternative is to set timers and not issue calls any faster than the limit, but that is unreliable as someone else may make calls on the org at the same time as you and then rate limiting can still occur.

For some purposes you can use action batches, giving higher throughput...

https://developer.cisco.com/meraki/api-v1/action-batches-overview/

***if I am doing something that will generate high call volumes, I generally split it into smaller chunks to avoid just hammering on the org.

[ls08]

I really wouldn't suggest changing the code to anyone. Requests Retry-After will give you the identical results. I've built all my python codes using Meraki library and when I tried converting it to Requests with the IF statement I still get the same error 429 with a Retry-After in 59 or 60 seconds. Nothing greater or less and I had a max retry of 10. I thought  the Retry-After would give off random seconds, but it was consistently 59 or 60.

Because of this result, I'm adding time.sleep() with a random integer in between API calls and use this setting for my dashboard.

dashboard = meraki.DashboardAPI(API_KEY, suppress_logging=True,wait_on_rate_limit=True,maximum_retries=10)

Supposedly Meraki is looking into increasing the API calls to 50, but this was in early Alpha stages when I asked my rep.

[Oren]

Question to the group. When you hit 429, how do you investigate/troubleshoot/workaround it?