LACP problem between Meraki MS and Fortigate

[RafaelKelles]

We have a Stack - 1 (SW-ACESSO2, SW-ACESSO3, SW-ACESSO1) and we are using port 47 from sw1 and sw2 to lacp to one Fortigate (port 1 and port 2). 

It seems to work well for some days but we lost connectivity Stack - 1 (SW-ACESSO2, SW-ACESSO3, SW-ACESSO1) lost connectivity on Aug 13 from 00:46 to 07:15 (UTC-3).

The way to get connectivity to be restored was changing LACP from Fortigate side (add/del one of two ports).

So, I would like to confirm that LACP is properly configured on both sides.

LACP on Meraki side:

Aggregation group AGGR/0 (SW-ACESSO1 47 and SW-ACESSO2 47)
Port status Enabled
Type Trunk
Native VLAN 1
Allowed VLANs 2-4094
Access policy Open
Link negotiation Auto negotiate
RSTP Enabled
Port schedule Unscheduled
Port isolation Disabled
Trusted DAI Disabled
UDLD Alert only
Tags none
PoE Enabled
Port mirroring Not mirroring traffic

LACP Fortigate side:

FW-100F-KAPLAN-MATRIZ-RS (LAG-LAN-TELECOM) # show

config system interface

    edit "LAG-LAN-TELECOM"

        set vdom "root"

        set allowaccess ping snmp

        set type aggregate

        set member "port1 port2"

        set device-identification enable

        set device-user-identification disable

        set lldp-reception enable

        set lldp-transmission enable

        set monitor-bandwidth enable

        set snmp-index 11

    next

end

From Fortigate, we can see

distribution algorithm: L4
LACP mode: active
LACP speed: slow
LACP HA: enable

LACP flags: (A|P)(S|F)(A|I)(I|O)(E|D)(E|D)
(A|P) - LACP mode is Active or Passive
(S|F) - LACP speed is Slow or Fast
(A|I) - Aggregatable or Individual
(I|O) - Port In sync or Out of sync
(E|D) - Frame collection is Enabled or Disabled
(E|D) - Frame distribution is Enabled or Disabled

status: up
LACP state: established
actor state: ASAIEE
actor port number/key/priority: 2 17 255
partner state: ASAIEE

Is there anything wrong with this config?

Regards, Rafael.

[cmr]

@RafaelKelles I would change this section to include VLAN 1 in the allowed VLANs

Native VLAN 1
Allowed VLANs 2-4094

I also don't see what is allowed from the FortiGate side, can you please detail?

[RafaelKelles]

Hi cmr,

I got FGT config for VLAN on LAG.

Regards, Rafael.

=======================================

edit "LAG-LAN-TELECOM"
set vdom "root"
set allowaccess ping snmp
set type aggregate
set member "port1 port2"
set device-identification enable
set lldp-reception enable
set lldp-transmission enable
set monitor-bandwidth enable
set snmp-index 11
next
edit "SERVIDORES"
set vdom "root"
set ip 10.113.10.254 255.255.255.0
set allowaccess ping snmp
set alias "VLAN10"
set device-identification enable
set role lan
set snmp-index 12
set interface "LAG-LAN-TELECOM"
set vlanid 10
next
edit "DESKTOPS_20"
set vdom "root"
set ip 172.16.1.2 255.255.255.0
set allowaccess ping https snmp http
set alias "VLAN20"
set device-identification enable
set monitor-bandwidth enable
set role lan
set snmp-index 33
set interface "LAG-LAN-TELECOM"
set vlanid 20
next
edit "DESKTOPS_21"
set vdom "root"
set ip 10.113.21.254 255.255.255.0
set allowaccess ping snmp
set alias "VLAN21"
set device-identification enable
set role lan
set snmp-index 34
set interface "LAG-LAN-TELECOM"
set vlanid 21
next
edit "DESKTOPS_22"
set vdom "root"
set ip 10.113.22.254 255.255.255.0
set allowaccess ping snmp
set alias "VLAN22"
set device-identification enable
set role lan
set snmp-index 35
set interface "LAG-LAN-TELECOM"
set vlanid 22
next
edit "DESKTOPS_23"
set vdom "root"
set ip 10.113.23.254 255.255.255.0
set allowaccess ping snmp
set alias "VLAN23"
set device-identification enable
set role lan
set snmp-index 36
set interface "LAG-LAN-TELECOM"
set vlanid 23
next
edit "DESKTOPS_24"
set vdom "root"
set ip 10.113.24.254 255.255.255.0
set allowaccess ping snmp
set alias "VLAN24"
set device-identification enable
set role lan
set snmp-index 37
set interface "LAG-LAN-TELECOM"
set vlanid 24
next
edit "VOIP"
set vdom "root"
set ip 10.113.60.254 255.255.255.0
set allowaccess ping snmp
set alias "VLAN60"
set device-identification enable
set role lan
set snmp-index 38
set interface "LAG-LAN-TELECOM"
set vlanid 60
next
edit "WIFI CORP"
set vdom "root"
set ip 10.113.70.254 255.255.255.0
set allowaccess ping snmp
set alias "WIFI CORP"
set device-identification enable
set role lan
set snmp-index 39
set interface "LAG-LAN-TELECOM"
set vlanid 70
next
edit "WIFI GUEST"
set vdom "root"
set ip 10.113.80.254 255.255.255.0
set allowaccess ping snmp
set alias "WIFI GUEST"
set device-identification enable
set role lan
set snmp-index 40
set interface "LAG-LAN-TELECOM"
set vlanid 80
next
edit "IOT"
set vdom "root"
set ip 10.113.90.254 255.255.255.0
set allowaccess ping https ssh snmp http
set alias "VLAN90"
set device-identification enable
set role lan
set snmp-index 41
set interface "LAG-LAN-TELECOM"
set vlanid 90
next
edit "EQUIP_BIOME"
set vdom "root"
set ip 10.113.110.254 255.255.255.0
set allowaccess ping snmp
set alias "VLAN110"
set device-identification enable
set role lan
set snmp-index 42
set interface "LAG-LAN-TELECOM"
set vlanid 110
next
edit "LINK DED"
set vdom "root"
set ip 200.143.121.101 255.255.255.254
set allowaccess ping snmp
set description "link 100MB
circuit_id: PAE53005200624"
set alias "VLAN150 - BR DIGITAL"
set device-identification enable
set monitor-bandwidth enable
set role wan
set snmp-index 43
set interface "LAG-LAN-TELECOM"
set vlanid 150
next
edit "MPLS"
set vdom "root"
set allowaccess ping snmp
set description "LINK MPLS"
set alias "VLAN151"
set device-identification enable
set monitor-bandwidth enable
set role wan
set snmp-index 44
set interface "LAG-LAN-TELECOM"
set vlanid 151
next
edit "LINK ADSL"
set vdom "root"
set allowaccess ping snmp
set alias "VLAN152"
set device-identification enable
set monitor-bandwidth enable
set role wan
set snmp-index 45
set interface "LAG-LAN-TELECOM"
set vlanid 152
next
edit "VLAN200"
set vdom "root"
set ip 10.113.200.254 255.255.255.0
set allowaccess ping https snmp http
set alias "GERENCIA"
set device-identification enable
set role lan
set snmp-index 46
set interface "LAG-LAN-TELECOM"
set vlanid 200
next

[cmr]

@RafaelKelles it appears that the FGT only has VLANs 10,20,21,22,23,24,60,70,80,90,110,150,151,152 and 200.  I can't see a native VLAN on the trunk from that config either, so I'd match that on the Meraki.  Unless I'm reading it wrong!?!

[c_olson]

We have recently ran into issues with this using LACP from the FortiGate to Meraki MS350's and MS355's on firmware version 7.2.7, mostly just with FortiGate 80F's that we have identified so far. Sometimes one of the ports on the switches will just start displaying the following error, "Port running LACP and LACP has disabled this port". We have tried rebooting the switches and the firewall. We even reprovisioned firewalls. We had the message go away on one after re-provisioning it, factory reset/push config from fortimanager. However in each of these cases the firewall has been up and connected via LACP to these switches for quite some time, sometimes months without issues. We did notice some RSTP port status changes in the RSTP logs and also found out at one site that the site maintenance person had been rebooting the firewall if they lost internet connectivity without notifying anyone. We have this same template and configuration pushed out to over 40 facilities, but have just recently started hearing reports of these issues happening so we aren't sure of the cause yet at this point. Our Meraki side config in our scenario is native VLAN 10 and all VLANs allowed. On the firewall side we have the native VLAN untagged and then all used VLANs configured as VLAN interfaces on the LAG. Our firewall side shows the same with no errors or issues when we ran into the switch side reporting that one of the ports was disabled. We have a higher tier 3rd party support vendor assisting us with the rollout of these firewalls and this configuration and they stated they believe it may be a bug that only presents in rare scenarios and we are planning to start troubleshooting next week. I would love to hear if you find out any further information. 

[GIdenJoe]

Your pure aggregate config and flags seem correct.  VLAN config of course is another thing.  I recommend using an unused VLAN ID as native VLAN on the port leading to the FGT and then on the FGT not defining any IP on the aggregate itself but only on VLAN interfaces on that aggregate.

We do have a few customers that are running MS, MR but have an FGT as gateway and usually we do aggregates to those and have not yet run into this issue.

[c_olson]

I found that several versions of Meraki MS firmware release notes reference LACP errors causing issues and/or MAC address flapping on uplink issues. It looks like it’s a random occurrence thing and isn’t always repeatable. Another references a random occurrence of it happening on reboot or disconnect/reconnect. I believe that’s what we are seeing. It looks like version 17.1.1 is said to correct the issue.

[K2ACE]

SOLVED my issue:

Got An issue when LACP from FG to MS.  the PO (spanned across X1-x2) in the FG side has 5 subinterfaces and the meraki side has an aggregate interface and it was configured with VLAN 1-1000.   This configuration caused all the subints in the FG to be down (red).  When I modified the MS side in the aggregate interface to have only the VLANs required, like 10,20,30,40,50 and deleted the 1-1000, the Ints at the FG went up and have been steady ok for a while.

in other words, the Vlans numbers in the MS  should match the sub ints in the FG.

[My61]

We’re experiencing the same issue. Interestingly, it only seems to work when one of the aggregated ports is disabled.

When both ports are enabled as members, uplink load balancing is also activated. However, Meraki then blocks one of the uplinks with the following error:
"Port running LACP and LACP has disabled this port."

As a result, about half of our devices on the native VLAN 1 don't receive an IP address in the MGT network.

[My61]

I have created a ticket and Meraki support turned a feature on our network : Enforce LACP Active.

This solves the error for blocking ports. But still getting issues with load balancing. Looking into it.

[cmr]

At least that's a step in the right direction.  I do wish we could get a list of all these undocumented features...

[PhilipSannwald]

What (loadbalancing) algorithm did you set on your Fortigate LAG? The default L4?

You can check via CLI: "diagnose netlink aggregate name TheLAGsNameHere"

or "config system interface" "edit TheLAGsNameHere" "get" ("show" gives insufficient infos)

[My61]

Default L4 indeed. Maybe it is not the load balancing but it is still confusing that half of the devices in Meraki goes offline when turning on the second port member on the fortigate aggregate trunk.

[PhilipSannwald]

Indeed. If you check the diag output, your Fortigate will think the participating links are up, and balance accordingly (L4 = IP & Port).

Have you tried setting the Forti LAG algorithm to L2 or L3?

[My61]

Not yet, i will try it shortly.

[My61]

Well, Meraki Support tells me that our MS425 supports layer 2 link aggregation.

I am gonna try this on Monday...Thanks also so far.

[c_olson]

You just need to reach out to Meraki and have them enable visibility for the enforce LACP option in the GUI. Then turn the feature off and it will work as expected.

[PhilipSannwald]

This! "Enforce LACP Active" must be disabled on the Meraki LAG / AGGR.

Unless you unplug / disable the ports it may take up to 30s after the change was synchronized with the switches.

I changed the Fortigates' LAG's LACP Algorythm setting to "L2" as well. (edit LAGport // set algorithm L2 // end)

With this I am happy to report my FG120-HA-Cluster is playing nicely with the MS425-16 Stack acting as the core switch.

LAG1 connects Forti-A-X3&X4 to Meraki-A-1&B-1

LAG2 connects Forti-B-X3&X4 to Meraki-A-2&B-2

Since Forti-A und Forti-B are not connected to the same LAG on the Meraki Stack, there is no need for "set lacp-ha-secondary disable" / "set lacp-ha-slave disable".

[My61]

Thanks all. Yes it worked by disabling the feature.

[Jeff_Longley]

Following this as I'm just planning on something similar....

[c_olson]

You just need to reach out to Meraki and have them enable visibility for the Enforce LACP option. Then turn the feature off then Link Aggregation will work as expected once again. This started happening because Meraki added a new "hidden" feature that we don't have access to and enabled it by default on all newly created aggregation groups. 

[My61]

True it works!