Retrieving more than 2 pages of 1000 clients from GET /networks/{networkId}/clients?

[joincidence]

I posted on here about a week ago on how to get more than 1 page. The solution was looking at the headers.link and there are 2 links in there with relevant 'startafter' data to give me the pages. This gave me around 1800 clients, and I thought it would be sufficient. 

Now, I'm needing to go back further in time and there are more than 2000 clients in this wider time span. The problem is, after making this http GET, I'm still only seeing 2 links in the headers(despite there being roughly 3500 clients in the time span). So, I'm getting a total of exactly 2000 clients from 2 links. 

How do I find these extra pages, so I can get all of the users?

Here is the headers.link that gets returned(with my network id in 5s)

<https://n11.meraki.com/api/v0/networks/N_555555555555/clients?perPage=1000&startingAfter=a000000&tim...>; rel=first, <https://n11.meraki.com/api/v0/networ
ks/N_555555555555/clients?perPage=1000&startingAfter=k17a74c&timespan=604800>; rel=next

[CBurkhead]

You have to make repeated calls to the endpoint and each time, excluding the first call, specify the value for startingAfter. Each call with give you the token for the next "page" of data. That is why the Python code I posted checks for "rel=next" in the header and then updates the query string with the token it finds and keeps calling the endpoint until it no longer can find "rel=next", thus indicating that you have gotten the last page.

[joincidence]

I'm so close to wrapping my head around this, so bear with me if you can.

I just want to break this down to the simplest possible components, because I don't know Python(will eventually be in Powershell), and right now just using this: https://developer.cisco.com/meraki/api/#/rest/api-endpoints/clients/get-network-clients

So, I make the initial call for 1000 clients per page and I get this header.link

<https://n11.meraki.com/api/v0/networks/N_555555555555555/clients?perPage=1000&startingAfter=a000000&...>; rel=first, <https://n11.meraki.com/api/v0/networks/N_555555555555555/clients?perPage=1000&startingAfter=ka1710e&...>; rel=next

Now, I think I get the logic here. I'm supposed to now enter startingAfter=ka1710e in my next query, so that it starts at that page and gives me another, and so on. Problem is, when I do that, here's what I get next in the header.link

<https://n11.meraki.com/api/v0/networks/N_55555555555555555/clients?perPage=1000&startingAfter=a00000...>; rel=first, <https://n11.meraki.com/api/v0/networks/N_55555555555555555/clients?perPage=1000&startingAfter=k17db2...>; rel=next

This data confuses me. Shouldn't the first link be ka1710e and the second by something much further along than what it is?

[CBurkhead · Accepted solution]

OK, I will break this down the way I process it and keep the specific code out of it.

When I call my function, I pass it the URL for the endpoint (https://n11.meraki.com/api/v0/networks/N_555555555555555/clients) and the query information (timespan=30&perPage=1000)

The function starts a loop and calls the endpoint with this data. The first 1000 entries are returned. It saves that data. It then looks at the header.list and checks for "rel=next". If it finds it, it then gets whatever the token value is associated and then modifies the query to include it, using your example (timespan=30&perPages=1000&startingAfter=ka1710e)

It then goes back to the top of the loop, checks to see if it is done, and then calls the endpoint again with the new query values. It gets the next 1000 clients and concatenates the data to the end of the previous page. Then it checks for "rel=next" again. If it finds it, it gets the new token (k17db29) and updates the query to be timespan=30&perPages=1000&startingAfter=k17db29)

It goes to the top of the loop and sees it is not done. Calls the endpoint again with this query. Get the next page of data and concatenates it. Let's say this is the last page. It checks for "rel=next" and does not find it. It sets the flag to tell the loop it is done and returns to the top of the loop. The loop sees it is done and exits the loop. The function then returns all of the data it collected (3 pages worth) so it can be processed.

Does that make it a bit clearer?

[joincidence]

I see I see, ahh ok I get it now. Thanks for your help

[CBurkhead]

I am glad that helped. I had to look at the header.list data in a debugger as I stepped through the code and made calls to see the token changes and to see what I got after the first call, the next few calls, and then when I reached the end of the data. Being able to see that progression helped a lot in figuring out how they data was being presented and how to extract it.

[Qbradle]

Hello,

I'm not sure If anyone will see this but, I'm currently in a similar issue where I'm attempting to send a GET request that gathers all of my devices (I have over 2000) and I'm only getting 1000 max. I was wondering if you could post an example of the script you used. I'm new to python and I'm attempting to write a program that gathers all of my Meraki devices in my inventory and then have that saved to an excel sheet to document said data. Heres what I have so far:

import requests

import panda as a pd

api_key = "API_KEY"
headers = {
"X-Cisco-Meraki-API-Key": api_key
}

url = "https://api.meraki.com/api/v1/devices"
params = {
"perPage": 1000, # Number of devices per page (maximum)
"startingAfter": None # Pagination marker, initially set to None
}

all_inventory_data = []

while True:
response = requests.get(url, headers=headers, params=params)

if response.status_code == 200:
data = response.json()
all_inventory_data.extend(data)

if "nextPage" in response.headers:
# Retrieve the "startingAfter" value from the "nextPage" URL
params["startingAfter"] = response.headers["nextPage"].split("=")[-1]
else:
break
else:
print("Request failed with status code:", response.status_code)
break

print("Total devices retrieved:", len(all_inventory_data))

df = pd.DataFrame(data)

excel_filename = "meraki_Inventory(Used).xlsx"

df.to_excel(excel_filename, index=False, engine="openpyxl")                     # Saves the file to an excel sheet.

print("Data saved to", excel_filename)

[sungod]

I recommend use the Python library, it will do the page handling and many other things for you such as handling rate limiting - if you still want to write your own, you can see how the library code does it.

Install 'meraki' with pip...

https://pypi.org/project/meraki/

Project on github...

https://github.com/meraki/dashboard-api-python