Terraform cisco-open Meraki Provider Issues

[andrew_doering_]

Hello,

We have been trying to utilize the Terraform Meraki Provider, to help us programmatically obtain various information on our devices to utilize across other services. 

Generally, this goes alright, however, there are sometimes 2 major problems:

1. Getting Organization Information through a Data Lookup, results in failures and stops the flow for the rest of the information.

This happens randomly, and the code we are using is:

# Data source to retrieve all Meraki organizations
data "meraki_organizations" "orgs" {}

# Data source to retrieve all Meraki devices for the first organization
data "meraki_devices" "all_devices" {
organization_id = data.meraki_organizations.orgs.items[0].id
}

produces the following error in Meraki:

╷
│ Error: Failure when executing GetOrganizations
│ 
│ with data.meraki_organizations.orgs,
│ on network-zones-offices.tf line 20, in data "meraki_organizations" "orgs":
│ 20: data "meraki_organizations" "orgs" {}
│ 
│ error with operation GetOrganizations

I am not sure I understand the cause here, as the API Key is valid, and this error appears randomly. How can we future proof this and prevent it from occurring again? Of course we could hardcode an Org ID, but, we would prefer this information not be hardcoded so we can utilize it across multiple tenancies (if needed).

2. When the above doesn't fail, and everything works successfully, we have a massive TF file working, that obtains all of our IP Addresses for all of our Uplinks, but, we are trying to add Virtual IP Address information into that, and it seems to always fail with this error:

data "meraki_networks_appliance_warm_spare" "warm_spare" {
 for_each = toset([
for network_id in keys(local.appliances_by_network_id) :
 network_id if try(
 jsondecode(
 data.meraki_networks_appliance_warm_spare.warm_spare[network_id].item
 ).enabled,
false
 )
 ])
 network_id = each.key
}

data.meraki_networks_appliance_warm_spare.warm_spare["L_62*************808"]: Still reading... [3m20s elapsed]
data.meraki_networks_appliance_warm_spare.warm_spare["L_83*************708"]: Still reading... [3m20s elapsed]
data.meraki_networks_appliance_warm_spare.warm_spare["L_62*************858"]: Still reading... [3m10s elapsed]
data.meraki_networks_appliance_warm_spare.warm_spare["L_83*************021"]: Still reading... [3m20s elapsed]
data.meraki_networks_appliance_warm_spare.warm_spare["L_83*************709"]: Still reading... [3m10s elapsed]
data.meraki_networks_appliance_warm_spare.warm_spare["L_83*************720"]: Still reading... [3m20s elapsed]
data.meraki_networks_appliance_warm_spare.warm_spare["L_83*************662"]: Still reading... [3m20s elapsed]
...
...
...

╷
│ Error: Failure when executing GetNetworkApplianceWarmSpare
│ 
│   with data.meraki_networks_appliance_warm_spare.warm_spare["L_83*************709"],
│   on network-zones-offices.tf line 116, in data "meraki_networks_appliance_warm_spare" "warm_spare":
│  116: data "meraki_networks_appliance_warm_spare" "warm_spare" {
│ 
│ error with operation GetNetworkApplianceWarmSpare
╵

However, if I use my API Key and query against:

 https://api.meraki.com/api/v1/networks/L_83*************709/appliance/warmSpare

I get a valid response of:

{
"enabled": false,
"primarySerial": "OMITTED-INFO",
"spareSerial": null
}

However, this also occurs on a Network ID that does have a wamSpare enabled: 

https://api.meraki.com/api/v1/networks/L_62*************823/appliance/warmSpare

{
"enabled": true,
"primarySerial": "OMITTED-SERIAL-1",
"spareSerial": "OMITTED-SERIAL-2",
"uplinkMode": "virtual",
"wan1": {
"ip": "1.2.3.53",
"subnet": "1.2.3.48/29"
},
"wan2": {
"ip": "2.0.1.22",
"subnet": ""
}
}

(Note that the IP Addresses have been changed)

 Is there any known issues with the terraform provider being run on dashboards? I understand and am aware that it is in alpha/beta testing, but it has been working generally pretty well, except for these two problems.

[Oren]

Hi Andrew,
Can you kindly provide the provider's debug. This will help us get a clearer picture of the problem you are having.

To enable debug:

export MERAKI_DEBUG=true
export TF_LOG=DEBUG

[andrew_doering_]

Hey Oren, 

I have MERAKI_DEBUG=True already set, however I see no difference in the output. Is it case sensitive to true? 

I won't be able to provide the output of whole thing in public, as it also contains company specific information, which I can't disclose. Is it possible to DM the file to you as opposed to posting it in the forum? 

[Oren]

Sure.

which version are you using?
Can you try adding meraki_debug =“true” to the TF file?

provider "meraki" {
meraki_debug = "true"
}

[andrew_doering_]

I did try that, but don't see anything specifically that isn't already outputted by TF_LOG = Debug. 

Have DM'ed you the file contents.

We are using the latest release, 0.2.13.

[andrew_doering_]

EDIT; Now sent, sorry had some issues with the log file.

With that said, we had: 

GetNetworkApplianceWarmSpare

Work for 2 or 3 runs, and then it failed again. 

My assumption here is that this is due to API rate limiting issues.

[Oren]

That makes sense. Terraform is lightning quick and its default is 10 calls per second.
If you have other scripts and integrations running in the background - it can meet the API rate limit.

Can you try using the attribute `meraki_requests_per_second` and reduce the rate?
https://github.com/cisco-open/terraform-provider-meraki/blob/4527e00d8ebea783ce8bfe64979295de21fd434...

[andrew_doering_]

I tried this today, setting the provider to use 4 requests per second. 

meraki_requests_per_second = 4

The total elapsed time with the information I provided you (again apologies, I know it was a confusing mess in DMs) for the first run, took 5 minutes, at which point I believe it timed out and then again provided the error:

 Error: Failure when executing GetOrganizations
│ 
│ with data.meraki_organizations.orgs,
│ on network-zones-offices.tf line 15, in data "meraki_organizations" "orgs":
│ 15: data "meraki_organizations" "orgs" {}
│ 
│ error with operation GetOrganizations

So at this point, I don't think it is an API Rate Limiting issue anymore, as this was the first run of the day.

Does Meraki currently implement a cool down or wait API response header? Could that be utilized in the terraform code instead? Or is there any dashboard or use metrics in the Web UI that can be shown while doing this test so we can see if it is Rate Limiting or something else?

I don't think there is anything particularly complicated in our queries, just that we have 40+ offices, and there doesn't appear to be a way to easily limit the query or search further in the data resources.

For example, just now, I see a 429 error in Postman, which is much more helpful. Could this be relayed back as the error message in the Terraform plan/apply as opposed to the current errors?

[andrew_doering_]

Tried again, setting it to 2 requests per second, it began to work, and then when querying it started timing out when querying warm spares. It only failed on 5 network IDs. When trying to query the API with Postman at the same time I saw the "waiting for..." I did not receive 429s.

While admittedly it is a workaround, the workaround doesn't appear to be working in the current environment with Terraform (with our code anyways) in the current state.

Is there some other alternative that can be done? Or are there further optimizations we need to do with our Terraform code?