New tool: Action Batch Helper 👨🏼‍🎤⚡🐘

[John_on_API]

Action batches are incredibly powerful tools for streamlining bulk configurations, enabling you to combine up to 100 actions at a time into a single API call for speed and efficiency. 

But what if you need to submit 100,000 actions? You'd need to write some client logic to split those actions into 1,000 individual batches. And what if action 508 relies on action 444 succeeding? You'd need to submit them in a linear fashion, necessitating additional client logic, and to stop the batch submission if one of the earlier ones fails.

I've created a new Python module to make it easier to work with action batches at massive scale, handling these concerns for you when you have hundreds of thousands of configuration changes to make. 

Key features

• Creates discrete action batches of a custom size from an arbitrary list of actions
• Submits the batches only when there is room on the organization's action batch queue to run them
• Allows for simultaneous or linear batch submission, in cases where some of your actions rely on earlier actions succeeding
• Optionally generates a JSON preview file of the entire list of batches

Here's a demo of the batch helper in action, and a deep dive on the BatchHelper class and what it does. Or you can just jump to the code.

Happy coding!

[PhilipDAth]

What a lot of work.  Well done!

[Inderdeep]

@John_on_API : Good one !

[Greenberet]

Nice Work 😃

is there any reason, why you did this as a seperate project and not into the python library itself?

I thought about something similiar with the "(async) with" statement directly into the api.

when I take your example:

for network in target_networks:

    new_ssid_name = xp.generate_xkcdpassword(random_word_list, numwords=3)
    action1 = dashboard.batch.wireless.updateNetworkWirelessSsid(network['id'], 0, name=new_ssid_name)
    action_list_1.append(action1)

    new_network_name = xp.generate_xkcdpassword(random_word_list, numwords=5)
    action2 = dashboard.batch.networks.updateNetwork(network['id'], name=new_network_name)
    action_list_2.append(action2)

    action3 = dashboard.batch.networks.updateNetwork(network['id'], timeZone="America/Los_Angeles")
    action_list_3.append(action3)

all_actions.extend(action_list_1)
all_actions.extend(action_list_2)
all_actions.extend(action_list_3)

print(len(all_actions))

test_helper = batch_helper.BatchHelper(dashboard, organization_id, all_actions, linear_new_batches=True, actions_per_new_batch=50)

test_helper.prepare()
test_helper.generate_preview()
test_helper.execute()

it would become something like this:

with meraki.batch_helper(dashboard, organization_id, all_actions, linear_new_batches=True, actions_per_new_batch=50, generate_preview=True) as test_helper:
    for network in target_networks:
        test_helper.set_group(0) # should be executed in the first batch group
        new_ssid_name = xp.generate_xkcdpassword(random_word_list, numwords=3)
        dashboard.batch.wireless.updateNetworkWirelessSsid(network['id'], 0, name=new_ssid_name)

        test_helper.set_group(1) # should be executed after group 0 is finished
        new_network_name = xp.generate_xkcdpassword(random_word_list, numwords=5)
        dashboard.batch.networks.updateNetwork(network['id'], name=new_network_name)

        test_helper.set_group(2) # should be executed after group 1 is finished
        dashboard.batch.networks.updateNetwork(network['id'], timeZone="America/Los_Angeles")

normally I would have used 3 different batches with 3 loops, but I think a set_group (which would be a different list in the backend like in your example) looks definitely better.

I think it should also be possible to create different settings per group. e.g. running group 2 as parallel batches after group 1 has been finished.

the test_helper.prepare/generate_preview/execute methods would be called inside of BatchHelper.__a/exit__()


to make this work, we would have to update the generator config for the batches, so that they will do an internal call to test_helper.add_batch(my_batch)

[John_on_API]

You're reading my mind here -- having this part of the Python SDK is a no-brainer, but I want at least SOME user feedback before committing anything to the Python SDK. The more user feedback we can get (and adoption, interest, etc.) helps drive this.

[Greenberet]

additionally might be an iterator nice.

1) create the batch statements

2) (async) for x in test_helper.run_batches(batches)

     -> as soon as an item has finished it will return the result of the action

maybe a combination of the with and iterator syntaxes

with meraki.batch_helper(dashboard, organization_id, all_actions, linear_new_batches=True, actions_per_new_batch=50, generate_preview=True) as test_helper:
    for network in target_networks:
        test_helper.set_group(0) # should be executed in the first batch group
        new_ssid_name = xp.generate_xkcdpassword(random_word_list, numwords=3)
        dashboard.batch.wireless.updateNetworkWirelessSsid(network['id'], 0, name=new_ssid_name)

        test_helper.set_group(1) # should be executed after group 0 is finished
        new_network_name = xp.generate_xkcdpassword(random_word_list, numwords=5)
        dashboard.batch.networks.updateNetwork(network['id'], name=new_network_name)

        test_helper.set_group(2) # should be executed after group 1 is finished
        dashboard.batch.networks.updateNetwork(network['id'], timeZone="America/Los_Angeles")


(async) for x in test_helper:
    print(x.result)
# if an iterator isnt needed:
test_helper.execute()

[PhilipDAth]

Going async does speed things up alot.

[Guillaume6hat]

Great work, thank you !

I was interested in action batches, but at large scale, and it would have needed a lot of work to manage this on our side.

This is exactly what I wanted, and after a few tests, it seems to work as expected, and really simple.

Any update during the last 2 years ? Any plan to include in the python SDK ?

Thank you again

[John_on_API]

Hi @Guillaume6hat I'm really glad to hear that you've had a positive experience using the module and that it's worked as expected.

As far as integrating into the Python library, this is possible--could you speak to how it would improve your workflow? To what extent would you want to see it integrated?

[Guillaume6hat]

Hi @John_on_API 

This was really helpful to make everything faster.

To explain the initial need, we have a tool, working with API, to do Third Party VPN Reset. As this is not available through the dashboard (well, just a button would work, but... still nothing), and not available through APIs, we do some basic thing such as : remove the VPN tag from the network, wait 60 seconds for the change to be pushed on the device, and then add the VPN tag back.

This works, and is in place for some years now. 

Lately, we had some issues with the VPN partner, and we, sometimes, have to do a reset on multiple sites. So we needed a massive VPN Reset. 

First, I wanted to do the classic VPN reset, site by site, but it would have took the time to send an API call for each site, then wait 60 seconds, and then send again an API call for each site. Meaning, if we want to do it on 200 or 300 sites, it would take way longer than 60 seconds, and having a true impact on sites.

Then, I remembered the idea of action batches, but then, I realized using it was not that hard, but that I would need to split in multiple action batches, manage them, get the responses... because there could be more than 100 actions. 

With the helper, well, I just had to create my list of actions, create the helper, prepare and execute, and voila 🙂

Next step, I want to simplify the DRP, making it way faster.

Having it integrated would mean that it would be maintened, documented on the meraki.io page, also, no need to add another package, and I would be sure to be at the last version.

Thank you

[Guillaume6hat]

Hello @John_on_API ,

I have some doubts regarding some options, not sure to understand correctly.

I have a list of actions, that I then split in different batches per network. For example, per network  :

Create Vlan

Change DHCP option

Activate VPN

I want :

- each batch to run as asynchonous, because each action depends on the previous one, and if there is an error, it must be canceled.

- batches to run all at the same time, to make it faster, as there are not dependant on each other.

I use : 

batch_helper.BatchHelper(dashboard, orgID, batch_action_list,actions_per_new_batch=3,synchronous_new_batches=True)

linear_new_batches is set to False by default, so batches should run at the same time ?

confirmed_new_batches is set to True by default. It means  that as soon as the batch is submitted, it is started, right ? If I set to False, it will instead wait for all my batches to be submitted to start them ? With the limit of 5, better to let it to True I guess.

Is it right ?

Thank you for your time