Interesting MX84 issue w/port flapping, AnyConnect related?

[OVERKILL]

I have a ticket open with Meraki on this issue, but the run-down:

I have a site that I switched to AnyConnect near the end of last year, just using the default config, as they only have a couple of users, so default port of 443. 

Well, a few weeks ago, I get a call from them that their "internet is going up and down". Check the dashboard, see no immediate evidence of this happening (all green) and told her I'd upgrade them to 16.16, which I had scheduled, that night, and we'll see if that fixed it.

Well, we had one day of nothing happening and then it started again.Checked the logs more thoroughly this time and saw that all ports that had a link on them had flapped. Odd. I searched specifically for that and saw that this appeared to be happening at least once a day.I opened a ticket and they apparently saw some issues in the logs and told me they were sending me a new unit. Excellent I thought. 

So, installed the new unit and figured I'd check the logs to ensure that this error message was gone. It wasn't. I was seeing the exact same thing with the replacement unit:

Support didn't seem concerned because the customer hadn't complained yet, but I logged into the 2960 stack that's one of the units behind the MX and it was showing its uplink being connected/disconnected when I see the flap in the MX logs, so clearly, this is still actively taking place (GI1/0/1):

Then I noticed that the AnyConnect service seemed to be restarting at the same time:

Also, because of the use of the default, port, the AnyConnect service was getting hammered by foreign IP's and this appeared to be triggered it to restart:

 

So, while I wait on support to get back to me, I just changed the port AnyConnect was on to stop the hammering, and, so far, it has been 24hrs and no interface flap, but the real test will be tomorrow morning when all the users are back, as this may just be an unrelated correlation. 

However, IF this observation does indeed show that the hits on the service are triggering the interface flap, there is clearly an issue with the service that will have to be dealt with. 

[Make_IT_Simple]

If your MX wan interface is connected behind your 2960, check if the switch port has energy-efficient ethernet enabled, if it is, try disabling that feature and see if the MX port will still go down. Good luck

[OVERKILL]

I have 4x 2960's behind it, one as a 3x stack, one as a standalone for a separate network. There's also a cheap Trendnet PoE 100Mbit switch that runs an Aruba AP for guest WiFi on its own VLAN and it uses a rate limited WAN link on WAN2 at 10Mbit. 

None of the 2960's have energy efficiency enabled and the MX flaps ALL the active ports at once, which you can see in the screenshot, that includes both WAN links (which are connected to an Alcatel/Lucent ALU for fibre) and all three LAN links. 

This only started in February that I can see from the logs, the equipment has been the same for at least a year and AnyConnect was only enabled near the end of last year.

[Make_IT_Simple]

I don't see any connection between the two services. But, if really someone is sending too many requests to your MX on port 443, (based on what they are doing), your MX may run out of memory and eventually panic, which would explain AnyConnect Restarting and ports are flapping. If this is true, you shouldn't hear any complaints this week since you are using a different port. Also, you may set AnyConnect to use port 443 again but connect a switch between your ISP and the MX to mirror the uplink port of the MX and check what kind of traffic is sent to the MX on port 443. Please let us know what you find, this is really a strange problem.

[OVERKILL]

Will do, that's the purpose of this thread. I thought it was quite an interesting issue and I'm not seeing it on my two other MX84's, both with AnyConnect, but neither of them are getting hammered on 443 like this one was. Both have the same fibre service too, conveniently. 

I'll update tomorrow whether it happens in the morning or not. It seems to happen most frequently around 9:20-9:40AM based on the logs, but can also happen later in the day again. 

[OVERKILL]

Heard back from support this AM. @Make_IT_Simple you may find this amusing, they suggested the same thing as you about EEE being enabled. These are older 2960S models, EEE isn't available (the sh eee status command doesn't even work) and I advised them accordingly.

I also made it a point to mention the AnyConnect port change, as this AM? No flap. We'll see what the rest of the week brings but I've suggested they start looking there in terms of a potential cause. 

[Make_IT_Simple]

@OVERKILL   it looks like we are getting our information from the same source loool.

 I actually enabled AnyConnect in my personal MX using the default port and I noticed many hits in port 443 from random IPs. I know that Meraki has nothing to prevent this because the MX cannot differentiate between legit and malicious traffic since it is listening on port 443. It will eventually drop the malicious traffic but not before using part of the memory/CPU . Let's see what support will say about this.

[OVERKILL]

Yep, exactly, and it seems like this particular network is being targeted, at least that's my theory based on what I see in the logs for it vs my other MX84's with AnyConnect. 

Would be really nice with some form of geoblocking for this service. This particular unit only runs Enterprise, so it doesn't have any of the advanced protection stuff or L7. 

[Make_IT_Simple]

I have advanced security enabled and I am only allowing a couple of countries, but it only works outbound. Even the MX can drop inbound traffic for port 443, it would be difficult to know which IPs to block or allow.

[OVERKILL · Accepted solution]

Yep, I have Advanced Security on the other two MX84's (and my MX64), but this client went with the less expensive license. 

Meraki support got back to me and confirmed 100% CPU spikes that likely corresponded with the flaps and the hits to AnyConnect. Not sure what they will be able to do about it though. 

I'm seeing a few already even with the new port choice, though clearly nowhere near as many, so they must be running port scans. 

[Make_IT_Simple]

That makes perfect sense, the MX will drop everything if the CPU reaches 100% and I don't believe that Meraki can do anything about this (I hope that I am wrong). If you are able to confirm support statement, then you need to check where those requests are coming from and how many of them. If they are coming from a specific country or specific IPs, then you can just have the ISP black hole them. 

[OVERKILL]

Yes, I'll be reaching out to Rogers (the ISP) and seeing what they can do in terms of blocking this traffic. 

The few I traced were China, FWIW. 

[Make_IT_Simple]

Hopefully, all traffic is coming from one country and you guys don't have any business there. I one time had port forwarding enabled for testing and I did see some requests coming from at least 5 different countries. UK, China, France, Russia, Germany.

[OVERKILL]

Looks like two different countries (China, Russia). But the volume is just a trickle now compared to when it was on 443. I haven't seen any hits on it today for example. This client is a car dealership, so yeah, definitely no business in those countries, lol. 

Also, no further flaps of course, since AnyConnect isn't getting hammered now.