Network Traffic

duqa
Comes here often

Network Traffic

Hi all,

I need to retrieve via API the network traffic (bytes) generated from one ssid in one specific time interval (between t0 and t1, excellent if with a 1h resolution to manage the timezone). I tried to use "/networks/:networkId/wireless/usageHistory" but for me it is not good because it is mandatory the device serial (I have a lot of devices inside my network). The meraki support suggest me to use "/networks/:networkId/wireless/dataRateHistory" but honestly also using resolution => 300 the averageKbps is too big and I not know how convert the averageKbps to realm MB of traffic.

 

Someone can help me?

Thanks

12 Replies 12
rhbirkelund
Kind of a big deal
Kind of a big deal

I know that this may not be exactly what you were looking for, since it's not a single endpoint to call. But with manipulation, and usage of other endpoints, you can still get what you want.

 

#! /usr/bin/env python3

from datetime import datetime
import meraki

def ConvertKbpsToMbps(p_Kbps: int) -> float:
    """Convert Kbps to Mbps"""
    return p_Kbps/1000

def main():
    """Main function routine"""
    organization_id = ""
    network_id = ""

    TargetSsid = {
        "name": "Winona Router",
        "timespan": 86400, # Spanning over the last 24 hours
        "resolution": 3600 # 1 hour resolution
    }

    totalKbps = 0
    sentKbps = 0
    receivedKbps = 0

    dashboard = meraki.DashboardAPI()

    AllDevices = dashboard.organizations.getOrganizationInventoryDevices(
        organization_id, total_pages='all'
    )

    print(f"Getting all wireless devices for {network_id} ")
    AllWirelessDevices = []
    for device in AllDevices:
        if device['productType'] == "wireless" and device['networkId'] == network_id:
            AllWirelessDevices.append(
                {
                    "name": device['name'],
                    "serial": device['serial'],
                    "totalKbps": 0,
                    "sentKbps": 0,
                    "receivedKbps": 0,

                }
            )
    
    print("Finding the SSID number...")
    AllSsids = dashboard.wireless.getNetworkWirelessSsids(
        network_id
    )
    for ssid in AllSsids:
        if ssid['name'] == TargetSsid['name']:
            TargetSsid['number'] = ssid['number']
            break
    
    for ap in AllWirelessDevices:
        print("hold!")
        ApUsageHistory = dashboard.wireless.getNetworkWirelessUsageHistory(
            network_id,
            timespan=TargetSsid['timespan'],
            resolution=TargetSsid['resolution'],
            deviceSerial=ap['serial'],
            ssid=TargetSsid['number']
        )
        print(f"Summing usage history for AP {ap['name']}, over the last {TargetSsid['timespan']/.3600} hours.")
        for entry in ApUsageHistory:
            ap['totalKbps'] += entry['totalKbps']
            ap['sentKbps'] += entry['sentKbps']
            ap['receivedKbps'] += entry['receivedKbps']
    print("Done.")
    print()

    for ap in AllWirelessDevices:
        totalKbps += ap['totalKbps']
        sentKbps += ap['sentKbps']
        receivedKbps += ap['receivedKbps']
    print(f"Average usage [Mbps] for SSID {TargetSsid['name']}")
    print(f"\t Sent: {ConvertKbpsToMbps(sentKbps/(TargetSsid['timespan']/TargetSsid['resolution']))}")
    print(f"\t Received: {ConvertKbpsToMbps(receivedKbps/(TargetSsid['timespan']/TargetSsid['resolution']))}")
    print(f"\t Total: {ConvertKbpsToMbps(totalKbps/(TargetSsid['timespan']/TargetSsid['resolution']))}")
    
    print(f"Total usage [Mbps] for SSID {TargetSsid['name']}")
    print(f"\t Sent: {ConvertKbpsToMbps(sentKbps)}")
    print(f"\t Received: {ConvertKbpsToMbps(receivedKbps)}")
    print(f"\t Total: {ConvertKbpsToMbps(totalKbps)}")
    


if __name__ == "__main__":
    ScriptStart = datetime.now()
    main()
    print("\n$Time Elapsed:",datetime.now()-ScriptStart)
LinkedIn ::: https://blog.rhbirkelund.dk/

Like what you see? - Give a Kudo ## Did it answer your question? - Mark it as a Solution 🙂

All code examples are provided as is. Responsibility for Code execution lies solely your own.
duqa
Comes here often

Thanks a lot for your reply. If I understood your code, I need to call an API using the serial of APs. So if I have 30 APS I need to call 30 time the same API with different serial. Right?

 

Thanks

rhbirkelund
Kind of a big deal
Kind of a big deal

Since you can't get traffic for an entire network at a time, you'll have to poll if for a single AP at a time. yes. So you go into your inventory, and filter out all devices except for APs and the ones added to the Network of interrest, and then you poll usage per AP, and simply add it all together.

LinkedIn ::: https://blog.rhbirkelund.dk/

Like what you see? - Give a Kudo ## Did it answer your question? - Mark it as a Solution 🙂

All code examples are provided as is. Responsibility for Code execution lies solely your own.
duqa
Comes here often

The problem is that I have really large network (thousand of APs) and considering the rate limiting for me it is not possible (I will spend too much time). Thanks a lot for your support

rhbirkelund
Kind of a big deal
Kind of a big deal

Rate limiting is handled by the Meraki package, which is the one, I'm using. In my experience, it usually works fine.

LinkedIn ::: https://blog.rhbirkelund.dk/

Like what you see? - Give a Kudo ## Did it answer your question? - Mark it as a Solution 🙂

All code examples are provided as is. Responsibility for Code execution lies solely your own.
PhilipDAth
Kind of a big deal
Kind of a big deal

You can speed up @rhbirkelund's code a lot (maybe 10x faster) if you have that many APs by changing to using ayncio.

 

https://github.com/meraki/dashboard-api-python#asyncio 

rhbirkelund
Kind of a big deal
Kind of a big deal

Feeling challenged by @PhilipDAth I've attempted to upgrade my script to use asyncio. It's my first attempt at using it, and it seems to work. But I feel there's something off with the numbers, and I can't quite put a finger on it.

If you have any suggestions as to what it could be, let me know.

 

[Updated Script]

 

#! /usr/bin/env python3

from datetime import datetime
import asyncio
import meraki
import meraki.aio

def ConvertKbytesToMbytes(p_Kbytes: int) -> float:
    """Convert KB/s to MB/s"""
    return p_Kbytes/1000

async def GetWirelessUsageHistory(p_dashboard: meraki.aio.AsyncDashboardAPI,p_network_id,p_Target,p_Ap):
    """https://developer.cisco.com/meraki/api-latest/#!get-network-wireless-usage-history"""
    
    totalKbytes = 0
    sentKbytes = 0
    receivedKbytes = 0

    try:
        print(f"Getting usage for AP {p_Ap['name']}")
        ApUsageHistory = await p_dashboard.wireless.getNetworkWirelessUsageHistory(
            p_network_id,
            timespan=p_Target['timespan'],
            resolution=p_Target['resolution'],
            deviceSerial=p_Ap['serial'],
            ssid=p_Target['number']
        )
    except meraki.AsyncAPIError as e:
        print(f"Meraki API error {e}")
        return -1
    except Exception as e:
        print(f"Some other exception {e}")
        return -2

    print(f"Summing usage history for AP {p_Ap['name']}, over the last {p_Target['timespan']/3600} hours.")
    print()
    '''Meraki uses wrong notation. 
    Documentation says output is kilobytes-per-second (KB/s), but response keys use kilobits-per-second (Kbps)'''
    for entry in ApUsageHistory:
        if entry['totalKbps'] == None:
            entry['totalKbps'] = 0
        if entry['sentKbps'] == None:
            entry['sentKbps'] = 0
        if entry['receivedKbps'] == None:
            entry['receivedKbps'] = 0
        totalKbytes += entry['totalKbps']
        sentKbytes += entry['sentKbps']
        receivedKbytes += entry['receivedKbps']
    return totalKbytes,sentKbytes,receivedKbytes
    
async def main():
    """Main function routine"""
    organization_id = ""
    network_id = ""

    TargetSsid = {
        "name": "Winona Router",
        "timespan": 86400, # Spanning over the last 24 hours
        "resolution": 300 # 1 hour resolution
    }

    totalKbytes = 0
    sentKbytes = 0
    receivedKbytes = 0

    dashboard = meraki.DashboardAPI()

    AllDevices = dashboard.organizations.getOrganizationInventoryDevices(
        organization_id, total_pages='all'
    )

    print(f"Getting all wireless devices for {network_id} ")
    AllWirelessDevices = []
    for device in AllDevices:
        if device['productType'] == "wireless" and device['networkId'] == network_id:
            AllWirelessDevices.append(
                {
                    "name": device['name'],
                    "serial": device['serial'],
                    "totalKbytes": 0,
                    "sentKbytes": 0,
                    "receivedKbytes": 0,

                }
            )
    TotalNumberOfAps = len(AllWirelessDevices)
    
    print("Finding the SSID number...")
    AllSsids = dashboard.wireless.getNetworkWirelessSsids(
        network_id,
    )
    for ssid in AllSsids:
        if ssid['name'] == TargetSsid['name']:
            TargetSsid['number'] = ssid['number']
            break

    async with meraki.aio.AsyncDashboardAPI(
        maximum_retries=5
    ) as aiodashboard:
        WirelessUsageHistoryTask = [GetWirelessUsageHistory(aiodashboard,network_id,TargetSsid,ap) for ap in AllWirelessDevices]
        for task in asyncio.as_completed(WirelessUsageHistoryTask):
            [ap_totalKbytes,ap_sentKbytes,ap_receivedKbytes] = await task
            totalKbytes += ap_totalKbytes
            sentKbytes += ap_sentKbytes
            receivedKbytes += ap_receivedKbytes

    print()
    print(f"Average usage [MB/s] for SSID {TargetSsid['name']} over {TotalNumberOfAps} APs")
    print(f"\t Sent: {ConvertKbytesToMbytes(sentKbytes/(TargetSsid['timespan']/TargetSsid['resolution']))} MB/s")
    print(f"\t Received: {ConvertKbytesToMbytes(receivedKbytes/(TargetSsid['timespan']/TargetSsid['resolution']))} MB/s")
    print(f"\t Total: {ConvertKbytesToMbytes(totalKbytes/(TargetSsid['timespan']/TargetSsid['resolution']))} MB/s")
    
    print(f"Total usage [MB/s] for SSID {TargetSsid['name']} over {TotalNumberOfAps} APs")
    print(f"\t Sent: {ConvertKbytesToMbytes(sentKbytes)} MB/s")
    print(f"\t Received: {ConvertKbytesToMbytes(receivedKbytes)} MB/s")
    print(f"\t Total: {ConvertKbytesToMbytes(totalKbytes)} MB/s")
    return 0


if __name__ == "__main__":
    ScriptStart = datetime.now()
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())
    print("\n$Time Elapsed:",datetime.now()-ScriptStart)

 

 

LinkedIn ::: https://blog.rhbirkelund.dk/

Like what you see? - Give a Kudo ## Did it answer your question? - Mark it as a Solution 🙂

All code examples are provided as is. Responsibility for Code execution lies solely your own.
PhilipDAth
Kind of a big deal
Kind of a big deal

I can't see a problem - does it return different results to the non-asyncio version?

rhbirkelund
Kind of a big deal
Kind of a big deal

Depending on what I set resolution to, I get different usage's. The higher the resolution (300) the more usage, as compared to if I set it lower (3600).

 

I can't really wrap my head around what resolution changes, and how it affects usage.

LinkedIn ::: https://blog.rhbirkelund.dk/

Like what you see? - Give a Kudo ## Did it answer your question? - Mark it as a Solution 🙂

All code examples are provided as is. Responsibility for Code execution lies solely your own.
rhbirkelund
Kind of a big deal
Kind of a big deal

If you choose the run it, let me know how long it takes, and the number of APs that it runs over.

LinkedIn ::: https://blog.rhbirkelund.dk/

Like what you see? - Give a Kudo ## Did it answer your question? - Mark it as a Solution 🙂

All code examples are provided as is. Responsibility for Code execution lies solely your own.
duqa
Comes here often

OK. I will update you if I will use. Now I am investigating if I can use clients/usageHistories using the traffic of each clients in order to calculate the data volume. For this API I can use a bulk of macAddress, so the number of calls is less that call every single AP

sungod
Kind of a big deal

For overall usage I use...  https://developer.cisco.com/meraki/api-v1/#!get-network-clients  ...just call once per network.

 

Return data includes usage in KB, the SSID, you can filter the query to only include wireless, but I just grab everything.

 

It's not ideal as it only allows specifying a start time or a timespan (but not an end time), but as I have cron run it at the same time each day, with a 24 hour timespan, it's ok for my purposes.

 

Works ok across hundreds of networks, thousands of switches and APs, tens of thousands of clients.

 

Using the Python library with aio, on > 400 network with >20k clients in the last 24 hours, run time is less than two minutes.

 

 

Get notified when there are additional replies to this discussion.