New tool: Action Batch Helper 👨🏼‍🎤⚡🐘

[John-K]

Action batches are incredibly powerful tools for streamlining bulk configurations, enabling you to combine up to 100 actions at a time into a single API call for speed and efficiency. 

But what if you need to submit 100,000 actions? You'd need to write some client logic to split those actions into 1,000 individual batches. And what if action 508 relies on action 444 succeeding? You'd need to submit them in a linear fashion, necessitating additional client logic, and to stop the batch submission if one of the earlier ones fails.

I've created a new Python module to make it easier to work with action batches at massive scale, handling these concerns for you when you have hundreds of thousands of configuration changes to make. 

Key features

• Creates discrete action batches of a custom size from an arbitrary list of actions
• Submits the batches only when there is room on the organization's action batch queue to run them
• Allows for simultaneous or linear batch submission, in cases where some of your actions rely on earlier actions succeeding
• Optionally generates a JSON preview file of the entire list of batches

Here's a demo of the batch helper in action, and a deep dive on the BatchHelper class and what it does. Or you can just jump to the code.

Happy coding!

[PhilipDAth]

What a lot of work.  Well done!

[Inderdeep]

@John-K : Good one !

[Greenberet]

Nice Work 😃

is there any reason, why you did this as a seperate project and not into the python library itself?

I thought about something similiar with the "(async) with" statement directly into the api.

when I take your example:

for network in target_networks:

    new_ssid_name = xp.generate_xkcdpassword(random_word_list, numwords=3)
    action1 = dashboard.batch.wireless.updateNetworkWirelessSsid(network['id'], 0, name=new_ssid_name)
    action_list_1.append(action1)

    new_network_name = xp.generate_xkcdpassword(random_word_list, numwords=5)
    action2 = dashboard.batch.networks.updateNetwork(network['id'], name=new_network_name)
    action_list_2.append(action2)

    action3 = dashboard.batch.networks.updateNetwork(network['id'], timeZone="America/Los_Angeles")
    action_list_3.append(action3)

all_actions.extend(action_list_1)
all_actions.extend(action_list_2)
all_actions.extend(action_list_3)

print(len(all_actions))

test_helper = batch_helper.BatchHelper(dashboard, organization_id, all_actions, linear_new_batches=True, actions_per_new_batch=50)

test_helper.prepare()
test_helper.generate_preview()
test_helper.execute()

it would become something like this:

with meraki.batch_helper(dashboard, organization_id, all_actions, linear_new_batches=True, actions_per_new_batch=50, generate_preview=True) as test_helper:
    for network in target_networks:
        test_helper.set_group(0) # should be executed in the first batch group
        new_ssid_name = xp.generate_xkcdpassword(random_word_list, numwords=3)
        dashboard.batch.wireless.updateNetworkWirelessSsid(network['id'], 0, name=new_ssid_name)

        test_helper.set_group(1) # should be executed after group 0 is finished
        new_network_name = xp.generate_xkcdpassword(random_word_list, numwords=5)
        dashboard.batch.networks.updateNetwork(network['id'], name=new_network_name)

        test_helper.set_group(2) # should be executed after group 1 is finished
        dashboard.batch.networks.updateNetwork(network['id'], timeZone="America/Los_Angeles")

normally I would have used 3 different batches with 3 loops, but I think a set_group (which would be a different list in the backend like in your example) looks definitely better.

I think it should also be possible to create different settings per group. e.g. running group 2 as parallel batches after group 1 has been finished.

the test_helper.prepare/generate_preview/execute methods would be called inside of BatchHelper.__a/exit__()


to make this work, we would have to update the generator config for the batches, so that they will do an internal call to test_helper.add_batch(my_batch)

[John-K]

You're reading my mind here -- having this part of the Python SDK is a no-brainer, but I want at least SOME user feedback before committing anything to the Python SDK. The more user feedback we can get (and adoption, interest, etc.) helps drive this.

[Greenberet]

additionally might be an iterator nice.

1) create the batch statements

2) (async) for x in test_helper.run_batches(batches)

     -> as soon as an item has finished it will return the result of the action

maybe a combination of the with and iterator syntaxes

with meraki.batch_helper(dashboard, organization_id, all_actions, linear_new_batches=True, actions_per_new_batch=50, generate_preview=True) as test_helper:
    for network in target_networks:
        test_helper.set_group(0) # should be executed in the first batch group
        new_ssid_name = xp.generate_xkcdpassword(random_word_list, numwords=3)
        dashboard.batch.wireless.updateNetworkWirelessSsid(network['id'], 0, name=new_ssid_name)

        test_helper.set_group(1) # should be executed after group 0 is finished
        new_network_name = xp.generate_xkcdpassword(random_word_list, numwords=5)
        dashboard.batch.networks.updateNetwork(network['id'], name=new_network_name)

        test_helper.set_group(2) # should be executed after group 1 is finished
        dashboard.batch.networks.updateNetwork(network['id'], timeZone="America/Los_Angeles")


(async) for x in test_helper:
    print(x.result)
# if an iterator isnt needed:
test_helper.execute()

[PhilipDAth]

Going async does speed things up alot.