LACP problem between Meraki MS and Fortigate

[RafaelKelles]

We have a Stack - 1 (SW-ACESSO2, SW-ACESSO3, SW-ACESSO1) and we are using port 47 from sw1 and sw2 to lacp to one Fortigate (port 1 and port 2). 

It seems to work well for some days but we lost connectivity Stack - 1 (SW-ACESSO2, SW-ACESSO3, SW-ACESSO1) lost connectivity on Aug 13 from 00:46 to 07:15 (UTC-3).

The way to get connectivity to be restored was changing LACP from Fortigate side (add/del one of two ports).

So, I would like to confirm that LACP is properly configured on both sides.

LACP on Meraki side:

Aggregation group AGGR/0 (SW-ACESSO1 47 and SW-ACESSO2 47)
Port status Enabled
Type Trunk
Native VLAN 1
Allowed VLANs 2-4094
Access policy Open
Link negotiation Auto negotiate
RSTP Enabled
Port schedule Unscheduled
Port isolation Disabled
Trusted DAI Disabled
UDLD Alert only
Tags none
PoE Enabled
Port mirroring Not mirroring traffic

LACP Fortigate side:

FW-100F-KAPLAN-MATRIZ-RS (LAG-LAN-TELECOM) # show

config system interface

    edit "LAG-LAN-TELECOM"

        set vdom "root"

        set allowaccess ping snmp

        set type aggregate

        set member "port1 port2"

        set device-identification enable

        set device-user-identification disable

        set lldp-reception enable

        set lldp-transmission enable

        set monitor-bandwidth enable

        set snmp-index 11

    next

end

From Fortigate, we can see

distribution algorithm: L4
LACP mode: active
LACP speed: slow
LACP HA: enable

LACP flags: (A|P)(S|F)(A|I)(I|O)(E|D)(E|D)
(A|P) - LACP mode is Active or Passive
(S|F) - LACP speed is Slow or Fast
(A|I) - Aggregatable or Individual
(I|O) - Port In sync or Out of sync
(E|D) - Frame collection is Enabled or Disabled
(E|D) - Frame distribution is Enabled or Disabled

status: up
LACP state: established
actor state: ASAIEE
actor port number/key/priority: 2 17 255
partner state: ASAIEE

Is there anything wrong with this config?

Regards, Rafael.

[cmr]

@RafaelKelles I would change this section to include VLAN 1 in the allowed VLANs

Native VLAN 1
Allowed VLANs 2-4094

I also don't see what is allowed from the FortiGate side, can you please detail?

[RafaelKelles]

Hi cmr,

I got FGT config for VLAN on LAG.

Regards, Rafael.

=======================================

edit "LAG-LAN-TELECOM"
set vdom "root"
set allowaccess ping snmp
set type aggregate
set member "port1 port2"
set device-identification enable
set lldp-reception enable
set lldp-transmission enable
set monitor-bandwidth enable
set snmp-index 11
next
edit "SERVIDORES"
set vdom "root"
set ip 10.113.10.254 255.255.255.0
set allowaccess ping snmp
set alias "VLAN10"
set device-identification enable
set role lan
set snmp-index 12
set interface "LAG-LAN-TELECOM"
set vlanid 10
next
edit "DESKTOPS_20"
set vdom "root"
set ip 172.16.1.2 255.255.255.0
set allowaccess ping https snmp http
set alias "VLAN20"
set device-identification enable
set monitor-bandwidth enable
set role lan
set snmp-index 33
set interface "LAG-LAN-TELECOM"
set vlanid 20
next
edit "DESKTOPS_21"
set vdom "root"
set ip 10.113.21.254 255.255.255.0
set allowaccess ping snmp
set alias "VLAN21"
set device-identification enable
set role lan
set snmp-index 34
set interface "LAG-LAN-TELECOM"
set vlanid 21
next
edit "DESKTOPS_22"
set vdom "root"
set ip 10.113.22.254 255.255.255.0
set allowaccess ping snmp
set alias "VLAN22"
set device-identification enable
set role lan
set snmp-index 35
set interface "LAG-LAN-TELECOM"
set vlanid 22
next
edit "DESKTOPS_23"
set vdom "root"
set ip 10.113.23.254 255.255.255.0
set allowaccess ping snmp
set alias "VLAN23"
set device-identification enable
set role lan
set snmp-index 36
set interface "LAG-LAN-TELECOM"
set vlanid 23
next
edit "DESKTOPS_24"
set vdom "root"
set ip 10.113.24.254 255.255.255.0
set allowaccess ping snmp
set alias "VLAN24"
set device-identification enable
set role lan
set snmp-index 37
set interface "LAG-LAN-TELECOM"
set vlanid 24
next
edit "VOIP"
set vdom "root"
set ip 10.113.60.254 255.255.255.0
set allowaccess ping snmp
set alias "VLAN60"
set device-identification enable
set role lan
set snmp-index 38
set interface "LAG-LAN-TELECOM"
set vlanid 60
next
edit "WIFI CORP"
set vdom "root"
set ip 10.113.70.254 255.255.255.0
set allowaccess ping snmp
set alias "WIFI CORP"
set device-identification enable
set role lan
set snmp-index 39
set interface "LAG-LAN-TELECOM"
set vlanid 70
next
edit "WIFI GUEST"
set vdom "root"
set ip 10.113.80.254 255.255.255.0
set allowaccess ping snmp
set alias "WIFI GUEST"
set device-identification enable
set role lan
set snmp-index 40
set interface "LAG-LAN-TELECOM"
set vlanid 80
next
edit "IOT"
set vdom "root"
set ip 10.113.90.254 255.255.255.0
set allowaccess ping https ssh snmp http
set alias "VLAN90"
set device-identification enable
set role lan
set snmp-index 41
set interface "LAG-LAN-TELECOM"
set vlanid 90
next
edit "EQUIP_BIOME"
set vdom "root"
set ip 10.113.110.254 255.255.255.0
set allowaccess ping snmp
set alias "VLAN110"
set device-identification enable
set role lan
set snmp-index 42
set interface "LAG-LAN-TELECOM"
set vlanid 110
next
edit "LINK DED"
set vdom "root"
set ip 200.143.121.101 255.255.255.254
set allowaccess ping snmp
set description "link 100MB
circuit_id: PAE53005200624"
set alias "VLAN150 - BR DIGITAL"
set device-identification enable
set monitor-bandwidth enable
set role wan
set snmp-index 43
set interface "LAG-LAN-TELECOM"
set vlanid 150
next
edit "MPLS"
set vdom "root"
set allowaccess ping snmp
set description "LINK MPLS"
set alias "VLAN151"
set device-identification enable
set monitor-bandwidth enable
set role wan
set snmp-index 44
set interface "LAG-LAN-TELECOM"
set vlanid 151
next
edit "LINK ADSL"
set vdom "root"
set allowaccess ping snmp
set alias "VLAN152"
set device-identification enable
set monitor-bandwidth enable
set role wan
set snmp-index 45
set interface "LAG-LAN-TELECOM"
set vlanid 152
next
edit "VLAN200"
set vdom "root"
set ip 10.113.200.254 255.255.255.0
set allowaccess ping https snmp http
set alias "GERENCIA"
set device-identification enable
set role lan
set snmp-index 46
set interface "LAG-LAN-TELECOM"
set vlanid 200
next

[cmr]

@RafaelKelles it appears that the FGT only has VLANs 10,20,21,22,23,24,60,70,80,90,110,150,151,152 and 200.  I can't see a native VLAN on the trunk from that config either, so I'd match that on the Meraki.  Unless I'm reading it wrong!?!

[c_olson]

We have recently ran into issues with this using LACP from the FortiGate to Meraki MS350's and MS355's on firmware version 7.2.7, mostly just with FortiGate 80F's that we have identified so far. Sometimes one of the ports on the switches will just start displaying the following error, "Port running LACP and LACP has disabled this port". We have tried rebooting the switches and the firewall. We even reprovisioned firewalls. We had the message go away on one after re-provisioning it, factory reset/push config from fortimanager. However in each of these cases the firewall has been up and connected via LACP to these switches for quite some time, sometimes months without issues. We did notice some RSTP port status changes in the RSTP logs and also found out at one site that the site maintenance person had been rebooting the firewall if they lost internet connectivity without notifying anyone. We have this same template and configuration pushed out to over 40 facilities, but have just recently started hearing reports of these issues happening so we aren't sure of the cause yet at this point. Our Meraki side config in our scenario is native VLAN 10 and all VLANs allowed. On the firewall side we have the native VLAN untagged and then all used VLANs configured as VLAN interfaces on the LAG. Our firewall side shows the same with no errors or issues when we ran into the switch side reporting that one of the ports was disabled. We have a higher tier 3rd party support vendor assisting us with the rollout of these firewalls and this configuration and they stated they believe it may be a bug that only presents in rare scenarios and we are planning to start troubleshooting next week. I would love to hear if you find out any further information. 

[GIdenJoe]

Your pure aggregate config and flags seem correct.  VLAN config of course is another thing.  I recommend using an unused VLAN ID as native VLAN on the port leading to the FGT and then on the FGT not defining any IP on the aggregate itself but only on VLAN interfaces on that aggregate.

We do have a few customers that are running MS, MR but have an FGT as gateway and usually we do aggregates to those and have not yet run into this issue.

[c_olson]

I found that several versions of Meraki MS firmware release notes reference LACP errors causing issues and/or MAC address flapping on uplink issues. It looks like it’s a random occurrence thing and isn’t always repeatable. Another references a random occurrence of it happening on reboot or disconnect/reconnect. I believe that’s what we are seeing. It looks like version 17.1.1 is said to correct the issue.