cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Python Get API randomly hits limit

SOLVED
Highlighted
Here to help

Python Get API randomly hits limit

Hi,

working on a script for Meraki to count all wireless clients in a organization and there capabilities. But the code run randomly in to the API limit which is 5 per second. As a test is added allot of sleep timer to slow down the code but even that doesn’t work and i cannot figure out why.

 

 

import requests
import time
import os.path


headers = {'X-Cisco-Meraki-API-Key': xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'}
se = requests.session()

orgid = {'Test': '111111'}

cnt_80211b = 0
cnt_80211n = 0
cnt_80211ac = 0
cnt_80211ax = 0


try:
    timer = 0
    client_info = []
    for org in orgid:
        networks = 'https://api.meraki.com/api/v0/organizations/{}/networks'.format(str(orgid[org]))
        time.sleep(1)
        networks_data = se.get(networks, headers=headers).json()
        time.sleep(1)
        response = requests.request("GET", networks, headers=headers)
        time.sleep(1)
        print ("networks_data")        
        if response.status_code == 200:
            time.sleep(1)
            if networks_data == 'errors':
                time.sleep(5)
            else:
                for each in networks_data:
                    print(".....................................................")
                    print (each['name'])
                    print(".....................................................")
                    time.sleep(1)
                    clients = 'https://api.meraki.com/api/v0/networks/{}/clients'.format(str(each['id']))
                    clients_data = se.get(clients, headers=headers).json()
                    time.sleep(1)
                    response = requests.request("GET", clients, headers=headers)
                    time.sleep(1)                    
                    if response.status_code == 200:
                        time.sleep(1)
                        for each_cli in clients_data:
                            if clients_data == 'errors':
                                time.sleep(5)                    
                            else:
                                if each_cli['ssid'] is not None:
                                        time.sleep(1)
                                        clientsurl = 'https://api.meraki.com/api/v0/networks/{}/clients/{}'.format(str(each['id']), str(each_cli['id']))
                                        clientsdata = se.get(clientsurl, headers=headers).json()
                                        time.sleep(1)
                                        response = requests.request("GET", clientsurl, headers=headers)
                                        time.sleep(1)
                                        if response.status_code == 200:
                                            if clientsdata == 'error':
                                                print ("Error")
                                                print (clientsdata)
                                                time.sleep(5)                    
                                            else:
                                                time.sleep(0.3)
                                                if "802.11b" in clientsdata['wirelessCapabilities']:
                                                    print ("802.11b")
                                                    cnt_80211b = cnt_80211b + 1
                                                elif "802.11n" in clientsdata['wirelessCapabilities']:
                                                    print ("802.11n")
                                                    cnt_80211n = cnt_80211n + 1
                                                elif "802.11ac" in clientsdata['wirelessCapabilities']:
                                                    print ("802.11ac")
                                                    cnt_80211ac = cnt_80211ac + 1
                                                elif "802.11ax" in clientsdata['wirelessCapabilities']:
                                                    print ("802.11ax")
                                                    cnt_80211ax = cnt_80211ax + 1 
                                        elif response.status_code == 429:
                                            time.sleep((int(response.headers["Retry-After"]))+ 1)                               
                    elif response.status_code == 429:
                        time.sleep((int(response.headers["Retry-After"]))+ 1)
                        
        elif response.status_code == 429:
            time.sleep((int(response.headers["Retry-After"]))+ 1)
    print ("...............................")
    print ("802.11b")
    print (cnt_80211b)
    print ("...............................")

    print ("...............................")
    print ("802.11n")
    print (cnt_80211n)
    print ("...............................")

    print ("...............................")
    print ("802.11ac")
    print (cnt_80211ac)
    print ("...............................")

    print ("...............................")
    print ("802.11ax")
    print (cnt_80211ax)
    print ("...............................")

except Exception as ex:
    print ("ooopsss")
    print(str(ex))
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted
Here to help

Re: Python Get API randomly hits limit

Found the solution and it works like a charm. 

 

Meraki_get_wireless_clients_capabilities

View solution in original post

13 REPLIES 13
Highlighted
A model citizen

Re: Python Get API randomly hits limit

I'm sorry, but you'll have to fix that formatting of your python script, if you want help with it.
Highlighted
Here to help

Re: Python Get API randomly hits limit

Indeed thanks for the heads up!

 

Adjusted  

Highlighted
A model citizen

Re: Python Get API randomly hits limit

If have this in my code.

 

 

elif r.status_code == 429:
            print(f'Rate limited activated - Retrying after {r.headers["Retry-After"]}.')
            time.sleep(int(r.headers['Retry-After']))
            continue

So perhaps add the keyword continue.

 

Highlighted
Here to help

Re: Python Get API randomly hits limit

Thank you

 

Testing it now

Highlighted
A model citizen

Re: Python Get API randomly hits limit

Also, a bit of advice, use environment variables for your API key, so you don't have it stored directly in your code. Adds some security, and flexibility, so that others can use the same script, without having to modify the source code directly.

 

LIke this.

import os

[...]

API_KEY = os.environ.get('MERAKI_API_KEY', None) 
if API_KEY is None:
   raise SystemExit('Please set environment variable MERAKI_API_KEY')
else:
   API_KEY = os.environ['MERAKI_API_KEY']

 

Assumes you have created the env-var MERAKI_API_KEY on your system.

Highlighted
Here to help

Re: Python Get API randomly hits limit

tested and the code runs a bit longer but still after some time:

 

clients_data
{'errors': ['API rate limit exceeded for organization']}

 

 

 

Highlighted
Here to help

Re: Python Get API randomly hits limit

Thank you!

 

nice feature to add!

Highlighted
A model citizen

Re: Python Get API randomly hits limit

I think you should clean up your code, for starters.

Instead of mixing requests() and sessions() use only one.

 

Use functions, and segment your code, for simplicity.

def GetOrgs(p_apiKey):
    for _ in range(MAX_RETRIES):
        try: 
            r = session.get(
                baseURL+'/organizations',
                headers = {
                    'Authorization': f'Bearer {p_apiKey}',
                    'Content-Type': 'application/json',
                    "Accept": "application/json",
                }
            )
            if r.status_code == 200:
                return r
            elif r.status_code == 429:
                print(f'Rate limited - Retrying after {r.headers["Retry-After"]}.')
                time.sleep(int(r.headers['Retry-After']))
                continue
            else:
                raise SystemExit(f'Unexpected status code: {r.status_code} returned from server.')
            
        except Exception as e:
            pprint(e)

(Note: I'm using the v1 beta api..)

 

For every session.get(), check the status code.

If it's 429, put in a sleep.

 

It's not the script that's being rate limited, it's the request that's being throttled.

 

 

 

 

Highlighted
Here to help

Re: Python Get API randomly hits limit

totally agree on cleaning up the code and using functions. Not that good at Python but learning it on the go 🙂

 

do you perhaps have a complete code that i could use and modify?

 

 

Highlighted
A model citizen

Re: Python Get API randomly hits limit

Not for your specific case, unfortunately.

 

But perhaps you can modify this to siut your needs.

 

#!/usr/bin/env python3

from pprint import pprint
import os, time, sys
import argparse
import requests
from requests import Session
import json

MAX_RETRIES = 5
API_KEY = None

apiVersion = 'v1'
baseURL = 'https://api.meraki.com/api/'+apiVersion

class NoRebuildAuthSession(Session):
 def rebuild_auth(self, prepared_request, response):
   '''
   No code here means requests will always preserve the Authorization header when redirected.
   Be careful not to leak your credentials to untrusted hosts!
   '''
session = NoRebuildAuthSession()

def GetOrgs(p_apiKey):
    for _ in range(MAX_RETRIES):
        try: 
            r = session.get(
                baseURL+'/organizations',
                headers = {
                    'Authorization': f'Bearer {p_apiKey}',
                    'Content-Type': 'application/json',
                    "Accept": "application/json",
                }
            )
            if r.status_code == 200:
                return r
            elif r.status_code == 429:
                print(f'Rate limited - Retrying after {r.headers["Retry-After"]}.')
                time.sleep(int(r.headers['Retry-After']))
                continue
            else:
                raise SystemExit(f'Unexpected status code: {r.status_code} returned from server.')
            
        except Exception as e:
            pprint(e)

# Main routine function
def main(arg_org_id,arg_get_nw,arg_debug): 
    if arg_debug is not None:
         print(locals())
         print()

    API_KEY = os.environ.get('MERAKI_API_KEY', None) 
    if API_KEY is None:
        raise SystemExit('Please set environment variable MERAKI_API_KEY')
    else:
        API_KEY = os.environ['MERAKI_API_KEY']

    if arg_org is None:
        print('Getting list of organisations...')
        r = GetOrgs(API_KEY)
        rjson = r.json()
        print('Success!')
        pprint(rjson)

def run(args):
    parser = argparse.ArgumentParser()
    group = parser.add_mutually_exclusive_group()

    parser.add_argument('-o', '--org-id', dest='arg_org_id', help='Set the Organisation ID')
    group.add_argument('-n', '--get-networks', dest='arg_get_nw', help='Get Networks')
    parser.add_argument('-d', '--debug', dest='arg_debug')
    
    options = parser.parse_args(sys.argv[1:])
    main(**vars(options))

if __name__ == "__main__":
    run(sys.argv[1:])

Edit: Added sys package.

 

Highlighted
Here to help

Re: Python Get API randomly hits limit

Build a new script last night it works much better but still hits the API limit. 

😅

 

# This is a Python 3 script to count the total unique client MAC addresses connected to MR access points for 
#  an organization during the last month.
#
# Usage:
#  clientcount.py -k <api key> [-o <org name>]
#
# Parameters:
#  -k <api key>     :   Mandatory. Your Meraki Dashboard API key
#  -o <org name>    :   Optional. Name of the organization you want to process. Use keyword "/all" to explicitly
#                       specify all orgs. Default is "/all"
#
# Example:
#  clientcount.py -k 1234 -o "Big Industries Inc" 
#
# Notes:
#  * In Windows, use double quotes ("") to enter command line parameters containing spaces.
#  * This script was built for Python 3.7.1.
#  * Depending on your operating system, the command to start python can be either "python" or "python3". 
# 
# Required Python modules:
#  Requests     : http://docs.python-requests.org
#
# After installing Python, you can install these additional modules using pip with the following commands:
#  pip install requests
#
# Depending on your operating system, the command can be "pip3" instead of "pip".
#
# 

import sys, getopt, requests, json, time, datetime, os, sqlite3, threading
from ratelimit import limits


#SECTION: GLOBAL VARIABLES: MODIFY TO CHANGE SCRIPT BEHAVIOUR

#SECTION: GLOBAL VARIABLES AND CLASSES: DO NOT MODIFY

ARG_APIKEY                  = '' #DO NOT STATICALLY SET YOUR API KEY HERE
ARG_ORGNAME                 = '' #DO NOT STATICALLY SET YOUR ORGANIZATION NAME HERE
ORG_LIST                    = None #list of organizations, networks and MRs the used API key has access to
MAX_CLIENT_TIMESPAN         = 2592000 #maximum timespan GET clients Dashboard API call supports

        
class c_Net:
    def __init__(self):
        id          = ''
        name        = ''
        shard       = 'api.meraki.com'
        devices     = []
        
class c_Organization:
    def __init__(self):
        id          = ''
        name        = ''
        shard       = 'api.meraki.com'
        nets        = []
        
        
#SECTION: General use functions

#make it work nice across threads
def RateLimited(max_per_second):
  '''
  Decorator that make functions not be called faster than
  '''
  lock = threading.Lock()
  minInterval = 1.0 / float(max_per_second)
  def decorate(func):
    lastTimeCalled = [0.0]
    def rateLimitedFunction(*args):
      lock.acquire()
      elapsed = time.perf_counter() - lastTimeCalled[0]
      leftToWait = minInterval - elapsed

      if leftToWait>0:
        time.sleep(leftToWait)

      lock.release()

      ret = func(*args)
      lastTimeCalled[0] = time.perf_counter()
      return ret
    return rateLimitedFunction
  return decorate
    
def printhelp():
    print('This is a Python 3 script to count the total unique client MAC addresses connected to MR access points for')
    print(' an organization during the last month.')
    print('')
    print('Usage:')
    print(' clientcount.py -k <api key> [-o <org name>]')
    print('')
    print('Parameters:')
    print(' -k <api key>     :   Mandatory. Your Meraki Dashboard API key')
    print(' -o <org name>    :   Optional. Name of the organization you want to process. Use keyword "/all" to explicitly')
    print('                      specify all orgs. Default is "/all"')
    print('')
    print('Example:')
    print(' clientcount.py -k 1234 -o "Big Industries Inc"')
    print('')
    print('Notes:')
    print(' * In Windows, use double quotes ("") to enter command line parameters containing spaces.')
    
    
#SECTION: Meraki Dashboard API communication functions

@RateLimited(0.3)
def getInventory(p_org):
    #returns a list of all networks in an organization
    
    try:
        r = requests.get('https://api.meraki.com/api/v0/organizations/%s/inventory' % (p_org.id), headers={'X-Cisco-Meraki-API-Key': ARG_APIKEY, 'Content-Type': 'application/json'} )
    except:
        print('ERROR 06: Unable to contact Meraki cloud')
        return(None)
    
    if r.status_code != requests.codes.ok:
        print(f'Rate limited activated - Retrying after {r.headers["Retry-After"]}.')
        time.sleep(int(r.headers['Retry-After']))
        #return(None)
        raise Exception('API response: {}'.format(r.status_code))
        return response
    
    return(r.json())

@RateLimited(0.3)
def getNetworks(p_org):
    #returns a list of all networks in an organization
    try:
        r = requests.get('https://api.meraki.com/api/v0/organizations/%s/networks' % (p_org.id), headers={'X-Cisco-Meraki-API-Key': ARG_APIKEY, 'Content-Type': 'application/json'} )
    except:
        print('ERROR 07: Unable to contact Meraki cloud')
        return(None)
    
    if r.status_code != requests.codes.ok:
        print(f'Rate limited activated - Retrying after {r.headers["Retry-After"]}.')
        time.sleep(int(r.headers['Retry-After']))
        #return(None)
        raise Exception('API response: {}'.format(r.status_code))
        return response

    return(r.json())
        
@RateLimited(0.3)
def getNetworks_clients(p_org, p_dev):
    #returns a list of all networks in an organization
    try:
        r = requests.get('https://api.meraki.com/api/v0/organizations/%s/networks/%s/clients' % (p_org.id, p_dev), headers={'X-Cisco-Meraki-API-Key': ARG_APIKEY, 'Content-Type': 'application/json'} )
    except:
        print('ERROR 08: Unable to contact Meraki cloud')
        return(None)
    
    if r.status_code != requests.codes.ok:
        print(f'Rate limited activated - Retrying after {r.headers["Retry-After"]}.')
        time.sleep(int(r.headers['Retry-After']))
        #return(None)
        raise Exception('API response: {}'.format(r.status_code))
        return response
    
    return(r.json())

@RateLimited(0.3)  
def getNetworks_clients_id(p_org, p_dev, p_id):
    #returns a list of all networks in an organization
    try:
        r = requests.get('https://api.meraki.com/api/v0/organizations/%s/networks/%s/clients/%s' % (p_org.id, p_dev, p_id), headers={'X-Cisco-Meraki-API-Key': ARG_APIKEY, 'Content-Type': 'application/json'} )
    except:
        print('ERROR 08: Unable to contact Meraki cloud')
        return(None)
    
    if r.status_code != requests.codes.ok:
        print(f'Rate limited activated - Retrying after {r.headers["Retry-After"]}.')
        time.sleep(int(r.headers['Retry-After']))
        #return(None)
        raise Exception('API response: {}'.format(r.status_code))
        return response
    
    return(r.json())

@RateLimited(0.3)              
def getOrgs():
    #returns the organizations' list for a specified admin, with filters applied
    try:
        r = requests.get('https://api.meraki.com/api/v0/organizations', headers={'X-Cisco-Meraki-API-Key': ARG_APIKEY, 'Content-Type': 'application/json'} )
    except:
        print('ERROR 01: Unable to contact Meraki cloud')
        return(None)
    
    if r.status_code != requests.codes.ok:
        print(f'Rate limited activated - Retrying after {r.headers["Retry-After"]}.')
        time.sleep(int(r.headers['Retry-After']))
        #return(None)
        raise Exception('API response: {}'.format(r.status_code))
        return response
        
    rjson = r.json()
    orglist = []
    listlen = -1
    
    if ARG_ORGNAME.lower() == '/all':
        for org in rjson:
            orglist.append(c_Organization())
            listlen += 1
            orglist[listlen].id     = org['id']
            orglist[listlen].name   = org['name']
    else:
        for org in rjson:
            if org['name'] == ARG_ORGNAME:
                orglist.append(c_Organization())
                listlen += 1
                orglist[listlen].id     = org['id']
                orglist[listlen].name   = org['name']
    
    return(orglist)

@RateLimited(0.3)         
def getShardHost(p_org):
    #Looks up shard URL for a specific org. Use this URL instead of 'api.meraki.com'
    # when making API calls with API accounts that can access multiple orgs.
    #On failure returns None
    try:
        r = requests.get('https://api.meraki.com/api/v0/organizations/%s/snmp' % p_org.id, headers={'X-Cisco-Meraki-API-Key': ARG_APIKEY, 'Content-Type': 'application/json'} )
    except:
        print('ERROR 08: Unable to contact Meraki cloud')
        return None
    
    if r.status_code != requests.codes.ok:
        print(f'Rate limited activated - Retrying after {r.headers["Retry-After"]}.')
        time.sleep(int(r.headers['Retry-After']))
        #return(None)
        raise Exception('API response: {}'.format(r.status_code))
        return response
        
    rjson = r.json()
    
    return(rjson['hostname'])

@RateLimited(0.3)      
def refreshOrgList():
    global ORG_LIST
    cnt_80211b = 0
    cnt_80211n = 0
    cnt_80211ac = 0
    cnt_80211ax = 0
    cnt = 0
    print('INFO: Starting org list refresh at %s...' % datetime.datetime.now())
    
    orglist = getOrgs()
    if not orglist is None:
        for org in orglist:
            print('INFO: Processing org "%s"' % org.name)
            
            org.shard = 'api.meraki.com'
            orgshard = getShardHost(org)
            if not orgshard is None:
                org.shard = orgshard
            netlist = getNetworks(org)
            devlist = getInventory(org)
            
            for device in netlist:
                if device['name'] is not None:
                    print (device['name'])
                    clidev = device['id']
                    client_listlist = getNetworks_clients (org, clidev)
                    for client in client_listlist:                  
                        if client['ssid'] is not None:
                            client_id =  client['id']
                            client_spec = getNetworks_clients_id(org, clidev, client_id)
 
                            if "802.11b" in client_spec['wirelessCapabilities']:
                                cnt_80211b = cnt_80211b + 1
                            elif "802.11n" in client_spec['wirelessCapabilities']:
                                cnt_80211n = cnt_80211n + 1
                            elif "802.11ac" in client_spec['wirelessCapabilities']:
                                cnt_80211ac = cnt_80211ac + 1
                            elif "802.11ax" in client_spec['wirelessCapabilities']:
                                cnt_80211ax = cnt_80211ax + 1
                        
    print ("802.11b")
    print (cnt_80211b)
    print ("802.11n")
    print (cnt_80211n)
    print ("802.11ac")
    print (cnt_80211ac)
    print ("802.11ax")
    print (cnt_80211ax) 
    LAST_ORGLIST_REFRESH = datetime.datetime.now()      
    print('INFO: Refresh complete at %s' % LAST_ORGLIST_REFRESH)
                     
    return None
    
#SECTION: main
    
def main(argv):
    global ARG_APIKEY
    global ARG_ORGNAME
    
    #initialize command line arguments
    ARG_APIKEY      = ''
    ARG_ORGNAME     = ''
    arg_numresults  = ''
    arg_mode        = ''
    arg_filter      = ''    
    
    #get command line arguments
    try:
        opts, args = getopt.getopt(argv, 'hk:o:m:')
    except getopt.GetoptError:
        printhelp()
        sys.exit(2)
        
    for opt, arg in opts:
        if   opt == '-h':
            printhelp()
            sys.exit()
        elif opt == '-k':
            ARG_APIKEY      = arg
        elif opt == '-o':
            ARG_ORGNAME     = arg
        elif opt == '-m':
            arg_mode        = arg
            
    #check that all mandatory arguments have been given
    if ARG_APIKEY == '':
        printhelp()
        sys.exit(2)        
            
    #set defaults for empty command line arguments
    if ARG_ORGNAME == '':
        ARG_ORGNAME = '/all'
   
    refreshOrgList()
    
if __name__ == '__main__':
    main(sys.argv[1:])

 

Highlighted
A model citizen

Re: Python Get API randomly hits limit

The thing is, in the end, you can not avoid getting rate limited. If you submit more than 5 requests per second, it triggers the rate limit.

What you can do, is just be sure to handle it accordingly. That, or you can make sure to throttle your script manually but putting in sleep timers.

 

Or you can look into Action Batches

Highlighted
Here to help

Re: Python Get API randomly hits limit

Found the solution and it works like a charm. 

 

Meraki_get_wireless_clients_capabilities

View solution in original post

Welcome to the Meraki Community!
To start contributing, simply sign in with your Cisco account. If you don't yet have a Cisco account, you can sign up.