Air Marshal API calls (not all the expected events are returned)

[cta102]

I was asked if it would be possible to retrieve Air Marshal alerts via the API and  I thought that wouldn't be too much of a problem.

However the problem I encountered was that the python meraki.getairmarshal call only seems to retrieve the 'Other SSIDs' entries

For the Rogue SSIDs, Spoofs and Malicious Broadcasts and Packet floods categories no relevant data is returned.

I even ensured that I had Rogue SSIDs, Spoofs and Malicious Broadcasts both in the Air Marshal tab and in the logs.

Am I looking in the wrong place or are these values simply not available at this point in time?

One of the management teams are looking for an easy way to compare certain syslog server contents against data extracted from the Dashboard.

Rob...

[jdsilva]

Here's the doc for that call:

https://api.meraki.com/api_docs#list-air-marshal-scan-results-from-a-network

This call only returns scan results, not alerts. There isn't any other documented Air Marshal api calls 😞

[jdsilva]

Air Marshal events are available via Syslog... Not sure if that helps or not.

[jdsilva]

Wow, OK sorry for being Spammy McSpammerson...

But this was literally just added to the API about 20 mintues ago!

https://api.meraki.com/api_docs#list-the-events-for-the-network

So if you hit this API, and filter based on "includedEventTypes" for Air Marshl events:

Then I think you should be able to grab what you're looking for. I haven't tested this, as like I said, this API is hot off the press, but give it a go and let us know.

[cta102]

Thanks, that was a great bit of timing and gives me something to play with tomorrow (even though I'm on leave.)

Sadly another team looks after the syslog server and the problem is that I ended up going through a  spreadsheet containing events that their rules (and AI solution) had simply dumped to offline storage for being of no interest (including some deauth_bcast_attack events.)

I was called to check if the team were alerting properly for Meraki events and after I found ignored events I was asked asked if it would be possible to pull the Air Marshal alerts so the management team can double check against the directly sourced Meraki data. I also have no desire to work through another of their 106K row spreadsheets again

TBH it's also the sort of tool I would run for a while when a new contract is brought into service to confirm that everything aligns properly before anything is accepted into service.

[CBurkhead]

I am beginning to look at this endpoint and I am running into a problem getting the event log information. I found the event type I wanted by looking at /networks/{networkId}/events/eventTypes, which is "failover_event" (Primary uplink status change). When I am pulling the data, I am getting an error 400 with the message that:

"'includedEventTypes' must be an array"

Here is the code snippet that I am using for testing:

url = 'https://api.meraki.com/api/v0/networks/'+netId+'/events'
query = {'includedEventTypes':'failover_event'}
response = requests.request("GET", url, params=query, headers=header)

It does not matter if I specify the "failover_event" as a string, make it a Python list (['failover_event']), or integrate it into the URL itself and change the code to:

url = 'https://api.meraki.com/api/v0/networks/'+netId+'/events?includedEventTypes=failover_event'
response = requests.request("GET", url, headers=header)

I always get the error 400 and the same error message. any thoughts on how I can get around this and format the parameter correctly in Python?

[jdsilva]

@CBurkhead I read that as the syntax being:

query = {'includedEventTypes': ['failover_event',]}

But I get a 400 with that as well. I'm leaning towards something being broken here 😞

[CBurkhead]

OK, thanks for the sanity check @jdsilva! I read the details of that parameter as a Python list, too. Makes sense given that you could be specifying multiple events to include. 

It seems broken to me as well. Hopefully, that will be remedied soon. Getting access to the "failover_event" information will allow me to start collecting data via the API for a customer that wants some special reporting for their store locations.

[jdsilva]

***Edit nevermind, it's right there...

{'category': 'Appliance status', 'type': 'failover_event', 'description': 'Primary uplink status change'}

[CBurkhead]

No worries. I was about the post the list I got, which had the information as the second entry. Guess we go back to "its broken" and wait for Meraki to address the problem.

Love being on the bleeding edge of things. 🙂

[jdsilva]

@TonyC @chengineer @DexterLaBora Are you guys able to comment here on getting a 400?

[TonyC]

OP: from your most it sounds like you want alerts via the API. I suggest you take a look at webhooks for Airmarshal: https://developer.cisco.com/meraki/webhooks/#webhook-sample-alerts/sample3 first

Regarding the errors you're hitting, try including the ?productType=wireless parameter as well:
https://api.meraki.com/api/v0/networks/:networkId/events?productType=appliance&includesEventType=fai...

I'm verifying but I believe productType is actually a required parameter, and we need to update our docs to reflect that.

[CBurkhead]

I added in the productType and I am still getting the error 400. Verified the URL that is being constructed in requests and it looks correct to me. It comes back as:

https://n112.meraki.com/api/v0/networks/N_XXXXXXXXXXXXXXXXXX/events?productType=appliance&includedEv...

@jdsilva are you still getting the error?

[CBurkhead]

Update:

When I was trying this endpoint yesterday, before running into the issue with the includedEventTypes, I was getting an error 500. I was not sure why, but with the information from @TonyC, I know what was happening. It looks like the productType is required. If I specify only productType=appliance, I can pull the data from the event log. So, the following, works just fine.

url = 'https://api.meraki.com/api/v0/networks/'+netId+'/events'

query = {'productType':'appliance'}

response = requests.request("GET", url, params=query, headers=header)

You still have to do your own parsing if you are looking for particular events, but at least there is a method for getting the data.

[jdsilva]

I'm including productType and still getting a 400. 

[CBurkhead]

Well...I have good news and bad news. Good news first. I have gotten past the 400 error and am now getting data back from the endpoint using the includedEventTypes parameter. What needed to be done is the parameters sent to the endpoint needed to be dumped to JSON format, first. So, here is the updated line of code. BTW, this also works if you use data= instead of params=.

response = requests.request("GET", url, params=json.dumps(query), headers=header)

Now for the bad news. The data returned seems to totally ignore what ever event you specified for includedEventTypes. I was specifying failover_event and was getting DHCP and VPN events. I changed the event to client_connectivity and got the same events returned. Also, when specifying a perPage value, that is also ignored. I always got 10 events. They seem to be the 10 most recent events, regardless of the event type.

We might be a bit closer to being able to use this endpoint without doing our own parsing, but I think it is still broken.

[CBurkhead]

OK, I have some more data. If you don't use params= in the requests line, and use data= instead, you will get the same information back regardless of whether you do a json.dumps for the query. It is still not giving me the events I am asking for and not giving me the number of entries per page I specify, but I am also not getting the error 400.

I have also found that you don't need to specify the productType for a non-combined network if you do the json.dumps with params= or pass the query (with or without jason.dumps) using data=. Seems like that parameter really isn't required in this circumstance. 

@jdsilva are you seeing the same thing I am seeing? Do you just get the 10 most recent events for the network you are querying with no regard for the event(s) you asked for?

[jdsilva]

OK, wow this definitely needs some work still. 

If I do this, I get a 400 with the error: '{"errors":["productType is required"]}'

query = {'productType': 'appliance', 'includedEventTypes': ['failover_event']}
fails = requests.get(f"{base_url}/networks/{netid}/events", params=json.dumps(query), headers=headers)

But if I do this:

query = {'productType': 'appliance', 'includedEventTypes': ['failover_event']}
fails = requests.get(f"{base_url}/networks/{netid}/events?productType=appliance", params=json.dumps(query), headers=headers)

All is good and I get a 200 back with 10 events. But, as you say @CBurkhead I am getting a whole heck of a lot more than just failure events. In my case, the network I'm testing against is returning 10 DHCP Lease events. 

😞

[CBurkhead]

I am tending to see DHCP lease events and VPN events. All of those would be expected for the networks I am looking at. Have not been able to get past the 10 events. Have not tried asking for fewer than 10, but I bet I'd still get 10. 

[jdsilva]

Yup, just tried both 1000 events and 2 events. I got 10 both times. 

[CBurkhead]

Seems like all functionality for this endpoint beyond pulling all events is broken. Important safety tip, if you are grabbing all of the events, use the pageStartAt/pageEndAt data from each call to help determine how much data you want to pull. If you just grab it until there is no more you will end up with a huge amount of data. I think I stopped when I was initially looking at this when I was at 45 days back and 50,000+ records. Since there is no time period specified in the API call, you have to do it yourself.

[chengineer]

Hi guys, since you're doing this in Python, you may want to return the response.text error message to troubleshoot. In this case, I believe the 400 error you're seeing could be either "productType is required" or "'includedEventTypes'" must be an array". The former, which I believe you know this by now, is that for combined networks, you need to specify the productType for this endpoint to work.

For the latter error, if you're trying to filter for specific event types, that needs to be an array/list, so the way to do that using params within requests.get is to specify it as an array, namely 'includedEventTypes[]' as the key for your params dictionary.

Here's an example of a function that will do what you're looking for in Python using the requests library.

# https://api.meraki.com/api_docs#list-the-events-for-the-network
def get_event_log(api_key, networkId, productType=None, includedEventTypes=None, excludedEventTypes=None, deviceMac=None,
                  deviceSerial=None, deviceName=None, clientIp=None, clientMac=None, clientName=None, smDeviceMac=None,
                  smDeviceName=None, perPage=None, startingAfter=None, endingBefore=None, retries=5):
    resource = f'https://api.meraki.com/api/v0/networks/{networkId}/events'

    parameters = ['productType', 'deviceMac', 'deviceSerial', 'deviceName', 'clientIp', 'clientMac', 'clientName',
                  'smDeviceMac', 'smDeviceName', 'perPage', 'startingAfter', 'endingBefore']
    params = {key: value for (key, value) in locals().items() if key in parameters and value != None}
    if includedEventTypes:
        params['includedEventTypes[]'] = locals()['includedEventTypes']
    elif excludedEventTypes:
        params['excludedEventTypes[]'] = locals()['excludedEventTypes']

    while retries > 0:
        response = requests.get(resource, headers={'X-Cisco-Meraki-API-Key': api_key, 'Content-Type': 'application/json'}, params=params)

        if response.ok:
            print('success')
            return response.json()
        elif response.status_code == 429:
            wait = int(response.headers['Retry-After'])
            print(f'retrying in {wait} seconds')
            time.sleep(wait)
            retries -= 1
        else:
            message = response.text
            print(f'failed with error {response.status_code}, {message}')
            return None

[chengineer]

In other words, this API endpoint is working as intended, not broken, and just needs the right input data specified. Also, beware that the input for params should be a dict, not a string from json.dumps().

[CBurkhead]

Confirmed, this is now working. For anyone that does not want to dissect that Python code, the relevant information is the following:

query = {'includedEventTypes[]':['failover_event','dhcp_lease'],'perPage':'25'}
response = requests.request("GET", url, params=query, headers=header)

For includedEventTypes or excludedEventTypes, you need to put the [] after the keyword as part of the key. The parameters are given as a list. Make sure you also specify the productType if you are querying a combined network. This is not necessary if the network contains only one type of device.

[jdsilva]

OK, yup now that I've been schooled I got this working correctly as well. What I don't understand is what the difference is between  {'includedEventTypes[]': ['event_type']} and {'includedEventTypes': ['event_type']}, or when to use one over the other, so I clearly have some googling to do. 

Thanks for stepping in @chengineer ! Appreciate the clarification. 

[CBurkhead]

I have also never seen that structure before. Admittedly, I am still fairly new to Python, but I'd think that if it was something being used at all frequently, I'd have seen it at least once. I also spent some time on Google last night and I have yet to find a reference to this. @jdsilva if you find something, please point me in the right direction. 

At first glance it just looks like adding [] to the dictionary key value. But if that is all it is, then I don't see why you would not just use "includedEventTypes", as you said.

[jdsilva]

In my quick search last night I believe that the [] has something to do with how the URL is encoded, and not anything directly related to Python specifically. But I need to search a bit more. 

[chengineer]

Investigating this a bit more... the formatting here has to do with URL syntax for GET methods and params that are lists/arrays. For example, if we want to filter on two different types of events, the syntax for the GET URL would be something like includedEventTypes[]=dhcp_lease&includedEventTypes[]=client_connectivity. To do this in Postman, you would need to add two query params with the same key (screenshot below). However, since Python's params is a dict, the way to pass in the array using requests.get would be to pass in a list for the value, so 

params['includedEventTypes[]'] = ['dhcp_lease', 'client_connectivity']

Postman example

[chengineer]

One other update here is if you specify 1000 perPage results, but the Meraki database takes longer than a backend threshold to retrieve the data, you may only receive some number of results less than 1000. Navigating to the next page though will continue where left off. Just note that there can be situations where you may not receive the full number of perPage events requested if the query takes a while.

[CBurkhead]

That should not be an issue on my end. I already have this endpoint going through a function to pull all of the page data and concatenating it. Thanks for the information. It is good to know that the results could be affected by the backend processing.

[jdsilva]

I found this thread on Stack Overflow which might shed a little light on the difference between using the [] or not.

https://stackoverflow.com/questions/3061273/send-an-array-with-an-http-get

This person is stating that it depends on the language used on the server, whether it is a strong or weak typed language. If this is accurate then it's good information, but I'm not sure how to use this to determine which I should be using when I create code. 

[chengineer]

FYI team, we just released a complete update of the Python SDK/library where pagination and query array parameters are handled automatically. You can find more details here on this post in the Early Access Developer Program: https://community.meraki.com/t5/Developer-Early-Access-Resources/Python-SDK-entirely-revamped-amp-up...