MX NBAR blocking Statistical Peer-to-peer traffi

SOLVED
Dunky
Building a reputation

MX NBAR blocking Statistical Peer-to-peer traffi

Our MX has begun blocking outbound traffic as per below:

Dunky_0-1644249973572.png

In my L7 firewall rules we block All Peer-to-peer.

How do I retain the L7 rule but permit this specific traffic as as exception.

 

This is what I see against the client:

Dunky_1-1644250121612.png

 

Thanks an advance.

 

1 ACCEPTED SOLUTION
Dunky
Building a reputation

Update - I raised a call with Meraki and Development have applied a fix on the backend for me.

I wont share the Case # in public chat, but if anyone wants it please PM me.

Statistical P2P which was getting NBAR blocked is now working and the L7 Deny All Peer-to-Peer is still in place.

View solution in original post

44 REPLIES 44
MarkB2
Here to help

Unfortunately there is no way with Meraki to permit anything layer 7, only block. So we can't for example have a more specific permit rule for this traffic followed by the L7 deny. I ran into this limitation with Geo-Blocking...

 

Please send them some feedback indicating you would like to see this feature.

PhilipDAth
Kind of a big deal

Can you whitelist ID 1889 under Security & SD-WAN/Threat Protection?

No, you can't do that unfortunately. Funnily enough, Meraki is using some NBAR IDs that aren't documented within the standard NBAR docs. Guess your best option is getting in contact with Meraki TAC.

HScar
Here to help

We are also suffering from false positive blocks. I can't believe Meraki doesn't allow any kind of whitelisting on NBAR because it's not super accurate. +1 on the feature request. 

Dunky
Building a reputation

Thanks for the replies chaps.

So, to permit this one out, I have to remove the entire Peer-to-Peer category from my L7 rules and hence permit all the nasty stuff too.

Doesn't this defeat the point of a firewall?

Oh well, it is what it is, not very well thought out in my opinion, we should be able to whitelist specific traffic flows.

 

JohnJ7
New here

Agreed.  For us, blocking NBAR 1889 resulted in our Cisco Agent Desktop being blocked from communicating with the UCCX server.  Cisco against Cisco, sure way to beat yourself up.

lpopejoy
A model citizen

Ouch, this just bit me too.  

harmankardon
Getting noticed

Also having trouble with this, L7 peer-to-peer is blocking our remote support tool.

Yes, same here.  

MarcAEC
Getting noticed

For years, Meraki has received feedback about the problems the lack of whitelisting for geo-blocking causes.  So, not only did they ignore that feedback, but then decided to build the same problems into the NBAR and the layer 7 rules.  😫

 

The best work-around I could come up with was to remove the rule to "Deny All Peer-to peer (P2P) and replace it with 5 rules to block BitTorrent, DC++, eDonkey, Gnutella, and Kazaa.  It seems the problematic rule for our legitimate applications is "Encrypted P2P".

Dunky
Building a reputation

Update - I raised a call with Meraki and Development have applied a fix on the backend for me.

I wont share the Case # in public chat, but if anyone wants it please PM me.

Statistical P2P which was getting NBAR blocked is now working and the L7 Deny All Peer-to-Peer is still in place.

MarcAEC
Getting noticed

Are you sure they just didn't exclude "Statistical P2P" from the internal rules when "Deny All Peer-to-Peer" is selected, the same as we could do by creating individual P2P rules for the other categories?

Dunky
Building a reputation

The problem with creating individual Deny rules for each category is when new ones are added you may forget to go in and add Deny rules for them.

MarcAEC
Getting noticed

The real solution (which we won't get) would be for Meraki to implement an exclusion function.  Do we know what we're losing by having Statistical P2P off?  What does it protect against when it is not blocking false positives.

MarcAEC
Getting noticed

The disadvantage with support doing something behind-the-scenes is there's no visibility that "All P2P" is excluding the statistical P2P.  Your successor may not know that's what's going on.

Dunky
Building a reputation

Historically that traffic has always been permitted - until they started NBAR blocking it, so it's effectively just reverting to the previous behaviour.

Would be much better if Soltatistucal P2P was listed jnbthe drop down and you could have a L7 Allow rule.

Thank you!

I've been banging my head against the wall for 4 days now.

I've disable the layer 7 rule to prove to my boss that this is the issue. I've opened a case with Support and hope that I'll get whatever work around you have as well.

Hi Dunky, Meraki support is giving me the run around on the same issue, can you send me the Case # or explain what they did? They want me to turn off my P2P filtering as a solution, which i don't want to do. They are literally blocking traffic to a support server server in my possession calling it P2P.

lpopejoy
A model citizen

The resolution to this is very concerning.  The OP gave me his case #, and Meraki performed the "back end fix".  

 

However, support will not tell me what this fix does.  Our cyber security policy and change management requires documentation on what is being done.  How can I allow a 3rd party to make changes that I can't qualify or document?  

 

Support says:  It is against company policy to disclose backend changes. 

 

How would you guys respond to that?

I would say that's disappointing if that's their plan to resolve this.

 

As many have said, there should be an allow option for L7 rules and/or an allow option for NBAR categories in the dashboard.

CptnCrnch
Kind of a big deal

So you're also asking software vendors to provide you with the source code of bugfixes? 🤔

No, but if the software vendor's fix makes a material change to the functionality, that should be documented in release notes.  The most likely fix that support is implementing is to cause "statistical P2P" to be excluded when "All P2P" is selected.  If that's what they're doing, they should disclose that versus saying "don't worry, trust us".

Of course not.  But they didn't make a source code change, they made a configuration change.  

 

Any enterprise change control process would require documentation of changes made.  

 

Clearly the change disables something, what is it?

AlexP
Meraki Employee

If you have a case number, may I take a look? What's being described here is raising some significant concerns that I want to take a deeper look into

Dunky
Building a reputation

I will PM you Alex.

I am noticing that traffic to Azure is also getting blocked by this:

Dunky_0-1649317717684.png

 

What we need is a way of having ALLOW rules, and specifying src and dst.

And why is "Statisitcal Peer-to-Peer" not even in the dropdown??

 

 

AlexP
Meraki Employee


@Dunky wrote:

I will PM you Alex.

I am noticing that traffic to Azure is also getting blocked by this:

Dunky_0-1649317717684.png

 

What we need is a way of having ALLOW rules, and specifying src and dst.

And why is "Statisitcal Peer-to-Peer" not even in the dropdown??

 

I wish I had an answer I could give you in that regard, but it's not my place to comment on design decisions publicly.

I will say that that category is aggregated as part of the Encrypted Peer to Peer categories, so anything that tries to inspect encrypted traffic will include it. Since we have no way to inspect encrypted application payloads, NBAR tries to use heuristics instead, although unfortunately the documentation is rather lacking it what it does: https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/qos_nbar/prot_lib/config_library/pp5600/nbar-prot-...

lpopejoy
A model citizen

I will PM you with my case # as well.

Dunky
Building a reputation

I am of the firm opinion that this hasn't really been thought through before implementation.

Why suddenly start blocking traffic that was previously permitted without a means of us permitting the traffic its now blocking.

Any why on earth apply it to inter-VLAN - that has caused me weeks of trying to work out why certain apps were failing on the LAN

etb
Here to help

I found this thread while searching for leads on a new issue that has come up since our MX upgraded from 15.x to 16.16 (and hence NBAR was added). 

 

Legitimate LogMeIn remote control traffic started getting blocked intermittently under this same NBAR ID 1889 Statistical Peer to Peer.  All of our users are using LogMeIn the same way, but I see logs that only about 25% of them were blocked under this rule.  

 

I'm not sure if it's weird or to be expected that these blocks seemed to resolve on their own after like 3 mins to 30 mins.  (The users kept trying, and eventually it just worked.)  I did find in old NBAR release notes that this rule "Classifies traffic by comparing it to machine-learned clusters", so I'm not sure if the rule adapts in real time, or if the rule just misses some instances sometimes (and so my users were able to connect with LogMeIn on a random instance that NBAR missed).

 

I guess I'm going to give it a few hours to see how things go, and then probably need to remove the content filter for P2P if there are still issues.  Though obviously I'd rather not do that.

lpopejoy
A model citizen

The inverse was true for us.  Everything worked fine for a few days after the upgrade to 16.16, until suddenly it didn't...  

This. I had it running on one of our networks for 2 weeks with no issues. So I upgraded the next two offices and then the next two and then all of the sudden, wham... everyone is reporting issues.

AlexP
Meraki Employee

Hey folks, just to close my loop on this. A few people have sent me case numbers in which our Support team tried to make some behind the scenes changes to work around this, and after digging through some data on our end, I've confirmed conclusively that it does not work.

I will be sending out an internal PSA to them regarding this, but please reference this if you have a case going and they try to change things in a similar fashion - they'll know who I am, and can follow up with me internally if they have any questions.

MarcAEC
Getting noticed

Alex, if you have access to the product team, could you find out why they are so resistant to updating L7 to support exceptions.  This has been a big problem with geo-filtering for years.  When MaxMind decides 8.8.8.8 is in Iran, we can't put in a rule that exempts 8.8.8.8 from geo-filtering.  Instead, we have to allow all traffic from Iran!  

Any update on this Alex? We are still on 15.44 and getting pester messages to upgrade our devices.

I know nobody likes to hear "soon™" as an answer, but I can say that I'm aware of some positive developments regarding this.

Dunky
Building a reputation

Thank goodness for that.  Lets hope they add support for L7 ALLOW rules and specifying src &dst addresses.

That would address all the issues as far as I am concerned.

Erminio
Conversationalist

I have the same problem over here, out of the blue, problems arise on NBAR ID blocking.

Suddenly Google.docs didn't work any more, i have to change the L7 rule set, from block all Advertising to 15 different sub rules so i could allow Google Advertising. Now its happening with the voice devices who could not find the PBX in the datacenter (NBAR ID 1889). Hope the will find a fix for this problem soon.

dade80vr
Getting noticed

Same here, still having problems!

thunt
Here to help

Same.  Blocking our Connectwise Connect support product with NBAR ID 1889.  I had to break out each individual P2P rule as a line item in the firewall.  Pain in the A$$!  Plus if a new category gets added I won't be protected unless I check on it periodically. Just one more thing to do.  C'mon Meraki we need to allow an exception.

Dunky
Building a reputation

We need to be able to have an ALLOW and also src and dst fields.

It's so annoying when internal inter-vlan traffic is getting blocked by L7 NBAR.

Meraki just haven't thought this through !!

Erminio
Conversationalist

We have reverted back to 15.44 for approximately 70 MX's. Disabling the NBAR ID database didn't work (Under Network/General/Traffic Analysis change to "Basic: collect generic traffic categories". Under the surface it is still hitting the NBAR ID database with "false positive" while it is disabled. We are waiting for a fix to this NBAR ID problem.

HScar
Here to help

I guess I'll pile on the list here.

 

Awesome NBAR is blocking Google Chrome Sync on TCP 5228. Thinks it's Google Hangouts/Streaming Video Category on Layer 7. NBAR 1087.

 

Also blocking OpenDNS under NBAR ID 1087. UDP 53 and thinks it's Google Hangouts too. There's another thread on this.

 

Who designed this garbage?

MarcAEC
Getting noticed

Let's not forget that this problem is really an example of a deeper problem.  The Merkai development team likes to design functionality that has broad effects without including a way for customers to implement exceptions.  The NBAR problem would be much smaller if there was a way to create ALLOW rules for confirmed mis-identifications.  Instead, it's an all or nothing choice.  This is the same problem with the geographic filtering.  Need to access one specific site in Russia?  Guess what --you need to allow all inbound and outbound traffic from the entire country. 

Totally agree. I'm so disappointed in their current design in L7/NBAR Firewall.

 

Just a side note, I have had some success using FQDN's in Layer 3 to bypass country blocks.

Get notified when there are additional replies to this discussion.
Welcome to the Meraki Community!
To start contributing, simply sign in with your Cisco account. If you don't yet have a Cisco account, you can sign up.
Labels