Route tracking issue-WHat could be the issue for this.

Merakibud
Here to help

Route tracking issue-WHat could be the issue for this.

Issue.JPG

Oct 17 10:14:51 Route tracking Route connection change peer_type: gateway, peer: 192.168.101.9, connection_status: connected
Oct 17 10:13:54 Route tracking Route connection change peer_type: gateway, peer: 192.168.101.9, connection_status: disconnected
Oct 17 08:53:10 Route tracking Route connection change peer_type: gateway, peer: 192.168.101.9, connection_status: connected
Oct 17 08:52:39 Route tracking Route connection change peer_type: gateway, peer: 192.168.101.9, connection_status: disconnected
Oct 17 08:11:58 Route tracking Route connection change peer_type: gateway, peer: 192.168.101.9, connection_status: connected
Oct 17 08:11:34 Route tracking Route connection change peer_type: gateway, peer: 192.168.101.9, connection_status: disconnected

 

I am having Issue with the loosing connectivity to the FortiGate firewall as seen above. In my network All the branch traffic comes to this site and Internet traffic is forwarded though the static route to the FortiGate and to the  Internet through ISP gateway. what could be the issue for loosing connectivity.

17 Replies 17
RaphaelL
Kind of a big deal
Kind of a big deal

Is the Fortinet dropping ICMP packets ? The MX will send ICMP packets all the time and if the destination stops responding , it will give a log like that. 

 

If you can , run a packet capture and try to corelate the logs and what you observe in the packet capture. That might help.

Merakibud
Here to help

During the issue happening Fortinet is reachable but the Meraki Firewall is unreachable and no internet connectivity.

RaphaelL
Kind of a big deal
Kind of a big deal

Any logs on the Fortinet that could help ? Have you tried opening a ticket with Support ?

Merakibud
Here to help

yes getting logs from FortiGate. Have opened the ticket with meraki

 

Merakibud
Here to help

During the issue happening Fortinet is reachable but the Meraki Firewall is unreachable and no internet connectivity.

cmr
Kind of a big deal
Kind of a big deal

Why do you have the Meraki on the internet, yet send the traffic to the Fortinet?  Why not just have the Meraki behind the Fortinet?

Merakibud
Here to help

client want Fortinet behind the Meraki firewall.

cmr
Kind of a big deal
Kind of a big deal

So why is the Fortinet directly connected to the Internet as well, VPN?

Merakibud
Here to help

Meraki as providing the site to site vpn.

cmr
Kind of a big deal
Kind of a big deal

So why not just have:

 

LAN -> Fortinet -> Meraki -> Internet

Merakibud
Here to help

We needed all the branch traffic come to the Meraki and then to the Fortinet and to the internet.

we could have Meraki as VPN concentrator but since we have Meraki in routed mode we placed the fortinet behind the MEraki

DavNel
Conversationalist

its uncanny but we are having what seems to be the exact same issue with our Meraki and Fortigate.  Below is our setup, but essentially we see the same sort of messages in our MX95, it looses connection to our Fortigate and as a result all our spoke sites go down and headoffice looses internet connectivity. 

 

Been going on since Oct 12th, with about 5 flaps a day lasting about 30 seconds each, then it comes up on its own.

 

when it occurs fortigate is unable to ping mx95 but the fortigate itself still has internet access.

laptop plugged directly into a lan port on the MX95 is not able to ping the mx95 or 8.8.8.8 when the outage occurs but is able to ping the fortigate

 

Have you solved your problem yet?

 

 

Layout2.png

eventlog.jpg

DavNel
Conversationalist

It would seem out issue has been isolated to the Cisco Merake IDS (Intrusion Detection System).  The IDS is currently disabled and we haven’t had an outage since.

BNITM
Conversationalist

Hello,

could you confirm that disabling IDS solved this issue? We have a similar problem using MX95 with a couple of network outages a day (approx. 20 / week) showing the same route connection change messages that includes all of our other routers within the LAN, but typically not the uplink (which is not a Fortigate device but a Unifi Edgerouter). Generally, our topology is quite different using a warm spare and a MS225-48LP as root switch for approx. 50 Unifi switches, but I am desperately looking for any clues to this problem.

DavNel
Conversationalist

Yes disabling the IDS fixed the issue.  While it wasnt exactly the IDS that was the issue it was the most recent definition package for the IDS.  But disabling the IDS also then prevented the meraki from using that bad definition package.  There was likely something in it that was either bad or causing a false positive.  

MartinLL
Building a reputation

I cant remember which version of MX this occured on, but the snort engine was upgraded from SNORT2 to SNORT3 at some point. On MX95 and MX85 this is a known issue and you see the route tracking drops due to the snort engine crashing. When SNORT restarts the dataplane stops forwarding traffic and the outage lasts for about 1 minute.

 

You have two options. Disable IDS or contact support. They can manually downgrade your network from SNORT3 to SNORT2.

MLL
BNITM
Conversationalist

Hello, thanks for your replies. This is really interesting, especially @MartinLL 's SNORT investigation. After disabling IDS I had no route change events and network outages since 10 days. This is the longest period of time so far since we started working on the problem. Hopping that we got rid of it, I think.I can do without IDS.

Get notified when there are additional replies to this discussion.
Welcome to the Meraki Community!
To start contributing, simply sign in with your Cisco account. If you don't yet have a Cisco account, you can sign up.
Labels