MX NBAR blocking Statistical Peer-to-peer traffi

[Dunky]

Our MX has begun blocking outbound traffic as per below:

In my L7 firewall rules we block All Peer-to-peer.

How do I retain the L7 rule but permit this specific traffic as as exception.

This is what I see against the client:

Thanks an advance.

[MarkB2]

Unfortunately there is no way with Meraki to permit anything layer 7, only block. So we can't for example have a more specific permit rule for this traffic followed by the L7 deny. I ran into this limitation with Geo-Blocking...

Please send them some feedback indicating you would like to see this feature.

[PhilipDAth]

Can you whitelist ID 1889 under Security & SD-WAN/Threat Protection?

[CptnCrnch]

No, you can't do that unfortunately. Funnily enough, Meraki is using some NBAR IDs that aren't documented within the standard NBAR docs. Guess your best option is getting in contact with Meraki TAC.

[HScar]

We are also suffering from false positive blocks. I can't believe Meraki doesn't allow any kind of whitelisting on NBAR because it's not super accurate. +1 on the feature request. 

[Dunky]

Thanks for the replies chaps.

So, to permit this one out, I have to remove the entire Peer-to-Peer category from my L7 rules and hence permit all the nasty stuff too.

Doesn't this defeat the point of a firewall?

Oh well, it is what it is, not very well thought out in my opinion, we should be able to whitelist specific traffic flows.

[JohnJ7]

Agreed.  For us, blocking NBAR 1889 resulted in our Cisco Agent Desktop being blocked from communicating with the UCCX server.  Cisco against Cisco, sure way to beat yourself up.

[lpopejoy]

Ouch, this just bit me too.  

[harmankardon]

Also having trouble with this, L7 peer-to-peer is blocking our remote support tool.

[lpopejoy]

Yes, same here.  

[MarcAEC]

For years, Meraki has received feedback about the problems the lack of whitelisting for geo-blocking causes.  So, not only did they ignore that feedback, but then decided to build the same problems into the NBAR and the layer 7 rules.  😫

The best work-around I could come up with was to remove the rule to "Deny All Peer-to peer (P2P) and replace it with 5 rules to block BitTorrent, DC++, eDonkey, Gnutella, and Kazaa.  It seems the problematic rule for our legitimate applications is "Encrypted P2P".

[Dunky · Accepted solution]

Update - I raised a call with Meraki and Development have applied a fix on the backend for me.

I wont share the Case # in public chat, but if anyone wants it please PM me.

Statistical P2P which was getting NBAR blocked is now working and the L7 Deny All Peer-to-Peer is still in place.

[MarcAEC]

Are you sure they just didn't exclude "Statistical P2P" from the internal rules when "Deny All Peer-to-Peer" is selected, the same as we could do by creating individual P2P rules for the other categories?

[Dunky]

The problem with creating individual Deny rules for each category is when new ones are added you may forget to go in and add Deny rules for them.

[MarcAEC]

The real solution (which we won't get) would be for Meraki to implement an exclusion function.  Do we know what we're losing by having Statistical P2P off?  What does it protect against when it is not blocking false positives.

[MarcAEC]

The disadvantage with support doing something behind-the-scenes is there's no visibility that "All P2P" is excluding the statistical P2P.  Your successor may not know that's what's going on.

[Dunky]

Historically that traffic has always been permitted - until they started NBAR blocking it, so it's effectively just reverting to the previous behaviour.

Would be much better if Soltatistucal P2P was listed jnbthe drop down and you could have a L7 Allow rule.

[jrsilvius]

Thank you!

I've been banging my head against the wall for 4 days now.

I've disable the layer 7 rule to prove to my boss that this is the issue. I've opened a case with Support and hope that I'll get whatever work around you have as well.

[askinner]

Hi Dunky, Meraki support is giving me the run around on the same issue, can you send me the Case # or explain what they did? They want me to turn off my P2P filtering as a solution, which i don't want to do. They are literally blocking traffic to a support server server in my possession calling it P2P.

[lpopejoy]

The resolution to this is very concerning.  The OP gave me his case #, and Meraki performed the "back end fix".  

However, support will not tell me what this fix does.  Our cyber security policy and change management requires documentation on what is being done.  How can I allow a 3rd party to make changes that I can't qualify or document?  

Support says:  It is against company policy to disclose backend changes. 

How would you guys respond to that?

[HScar]

I would say that's disappointing if that's their plan to resolve this.

As many have said, there should be an allow option for L7 rules and/or an allow option for NBAR categories in the dashboard.

[CptnCrnch]

So you're also asking software vendors to provide you with the source code of bugfixes? 🤔

[MarcAEC]

No, but if the software vendor's fix makes a material change to the functionality, that should be documented in release notes.  The most likely fix that support is implementing is to cause "statistical P2P" to be excluded when "All P2P" is selected.  If that's what they're doing, they should disclose that versus saying "don't worry, trust us".

[lpopejoy]

Of course not.  But they didn't make a source code change, they made a configuration change.  

Any enterprise change control process would require documentation of changes made.  

Clearly the change disables something, what is it?

[AlexP]

If you have a case number, may I take a look? What's being described here is raising some significant concerns that I want to take a deeper look into

[Dunky]

I will PM you Alex.

I am noticing that traffic to Azure is also getting blocked by this:

What we need is a way of having ALLOW rules, and specifying src and dst.

And why is "Statisitcal Peer-to-Peer" not even in the dropdown??

[AlexP]

---

@Dunky wrote:

I will PM you Alex.

I am noticing that traffic to Azure is also getting blocked by this:

What we need is a way of having ALLOW rules, and specifying src and dst.

And why is "Statisitcal Peer-to-Peer" not even in the dropdown??

I wish I had an answer I could give you in that regard, but it's not my place to comment on design decisions publicly.

I will say that that category is aggregated as part of the Encrypted Peer to Peer categories, so anything that tries to inspect encrypted traffic will include it. Since we have no way to inspect encrypted application payloads, NBAR tries to use heuristics instead, although unfortunately the documentation is rather lacking it what it does: https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/qos_nbar/prot_lib/config_library/pp5600/nbar-prot-...

[lpopejoy]

I will PM you with my case # as well.

[Dunky]

I am of the firm opinion that this hasn't really been thought through before implementation.

Why suddenly start blocking traffic that was previously permitted without a means of us permitting the traffic its now blocking.

Any why on earth apply it to inter-VLAN - that has caused me weeks of trying to work out why certain apps were failing on the LAN

[DylanO]

Did you end up figuring out a solution to this by chance, or is that backend change that they made still working for you? We're having the issue now pretty often when using our remote support tool. 

[etb]

I found this thread while searching for leads on a new issue that has come up since our MX upgraded from 15.x to 16.16 (and hence NBAR was added). 

Legitimate LogMeIn remote control traffic started getting blocked intermittently under this same NBAR ID 1889 Statistical Peer to Peer.  All of our users are using LogMeIn the same way, but I see logs that only about 25% of them were blocked under this rule.  

I'm not sure if it's weird or to be expected that these blocks seemed to resolve on their own after like 3 mins to 30 mins.  (The users kept trying, and eventually it just worked.)  I did find in old NBAR release notes that this rule "Classifies traffic by comparing it to machine-learned clusters", so I'm not sure if the rule adapts in real time, or if the rule just misses some instances sometimes (and so my users were able to connect with LogMeIn on a random instance that NBAR missed).

I guess I'm going to give it a few hours to see how things go, and then probably need to remove the content filter for P2P if there are still issues.  Though obviously I'd rather not do that.

[lpopejoy]

The inverse was true for us.  Everything worked fine for a few days after the upgrade to 16.16, until suddenly it didn't...  

[jrsilvius]

This. I had it running on one of our networks for 2 weeks with no issues. So I upgraded the next two offices and then the next two and then all of the sudden, wham... everyone is reporting issues.

[AlexP]

Hey folks, just to close my loop on this. A few people have sent me case numbers in which our Support team tried to make some behind the scenes changes to work around this, and after digging through some data on our end, I've confirmed conclusively that it does not work.

I will be sending out an internal PSA to them regarding this, but please reference this if you have a case going and they try to change things in a similar fashion - they'll know who I am, and can follow up with me internally if they have any questions.

[MarcAEC]

Alex, if you have access to the product team, could you find out why they are so resistant to updating L7 to support exceptions.  This has been a big problem with geo-filtering for years.  When MaxMind decides 8.8.8.8 is in Iran, we can't put in a rule that exempts 8.8.8.8 from geo-filtering.  Instead, we have to allow all traffic from Iran!  

[jrsilvius]

Any update on this Alex? We are still on 15.44 and getting pester messages to upgrade our devices.

[AlexP]

I know nobody likes to hear "soon™" as an answer, but I can say that I'm aware of some positive developments regarding this.

[Dunky]

Thank goodness for that.  Lets hope they add support for L7 ALLOW rules and specifying src &dst addresses.

That would address all the issues as far as I am concerned.

[DylanO]

Hi Alex, 

Just wondering if there is any update on this as we are having the same issue. Unable to shadow using SCCM Config Manager Remote Control as its being classified as peer to peer traffic. Thank you!

[AlexP]

Yes, 16.16.4 introduced some changes to NBAR that showed a significant reduction in false positive rates in internal testing.

If you're still having issues, please raise a new support case, as we need to look at this from a fresh perspective to see what still might be going on.

[etb]

Thanks, AlexP.

Are there any fine folks in the community who previously had statistical p2p false positives and who could share their experience after upgrading to 16.16.4 and re-enabling the p2p rules?

We are a small company with only 1 MX.  Also, the issue seemed to randomly only affect maybe 25% of users, and only sometimes affect them.  Although everyone is using the same affected application (LogMeIn) in the same manner, some users were never affected, some users were affected 1 hour but not the next, some users were affected 1 day but not the next, etc.  So I cannot think of any way that I could create a reliable testbed to try re-enabling the p2p rules without potentially randomly preventing some users from being able to work.

[Dunky]

I cant see how that can be a config/NBAR blocking issue.

Is there anything in the Event Log when you filter by Layer 7?

If not then I think you are best opening a case with Meraki direct given its intermittent behaviour.

[jrsilvius]

We've tried 16.16.4 and 16.16.5 we get a very similar experience. Apps work until they don't. I found that I had to set all the p2p rules separately and allow the ones that Meraki were saying were NBAR issues. I'm hoping that the 17.x firmware fixes all of this.

[etb]

@Dunky thanks, but yes, the very hit-or-miss issues were logged as blocked as statistical p2p, and it stopped within minutes of removing the configured p2p rules.

@jrsilvius thank you.  i guess i'll continue to hold off on reinstating the p2p rules until the community has more consistently better results.

[jrsilvius]

Hey Alex, I still have a case open. I'm hoping there is some good news with 17.x. 16.16.4 and 16.16.5 didn't fix our issues.

[Dunky]

I'm waiting for a stable v17 release.

Not looking forward to deploying it at a production site though I must admit.

[AlexP]

Many changes in regard to 17.x are still being added to 16.x as well, so if you are still having issues, please bring it back up with Support, and try to focus on what's still specifically getting blocked, along with packet captures if, and at all possible

[Isack2230]

I will tell you we are on 17.10 and we see SCCM internally being blocked by L7 rule and being classified as Statistical Peer-to-Peer

[etb]

Thanks, @Isack2230 .  

For anyone else following, I did upgrade to 16.16.5, but as of now I have not reenabled the L7 firewall rules nor content filtering rules.  Based on the traffic analytics page, I can still see our remote access software getting classified as "statistical P2P" maybe about 40% of the time.  For a little while it was classified as "encrypted P2P", but for a couple weeks now it has been "statistical P2P".  (I am not aware of any changes to the traffic itself, so I'm guessing that the change in classification is a result of changes in NBAR.)

I have had a case open with Meraki Support for about a month now, and we have been communicating back and forth.  At least based on this forum thread, it seems that Meraki Support has the impression that this issue is resolved.  So if others are still seeing it unresolved as well, then you probably want to let Support know.

[Dunky]

Came across another issue last week with Avaya IP Office client trying to connect to the IP Office - Password was being rejected as invalid - surprise surprise I found traffic being blocked and classified as Statisitical Peer to Peer. (v16.16.4)

[DylanO]

We're on 16.16.5 and still being blocked. I was hoping 17.10 would fix it but based on other comments here, it does not. I will open another case with them. Do you mind sharing your case # so i can reference it in my support request? If not, no worries i will point them to this forum. 

[etb]

hey, sorry, I think I'd rather not share the case # just because I don't honestly know what info could or could not be gleaned or social engineered with the case #.  I'll PM you my Meraki Support Engineer's name though.  And yeah, I included permalinks to various posts in this thread for reference in my Support case.

[DylanO]

That's unfortunate, we were hoping our upgrade to 17.x soon would solve it. 

[Erminio]

I have the same problem over here, out of the blue, problems arise on NBAR ID blocking.

Suddenly Google.docs didn't work any more, i have to change the L7 rule set, from block all Advertising to 15 different sub rules so i could allow Google Advertising. Now its happening with the voice devices who could not find the PBX in the datacenter (NBAR ID 1889). Hope the will find a fix for this problem soon.

[dade80vr]

Same here, still having problems!

[thunt]

Same.  Blocking our Connectwise Connect support product with NBAR ID 1889.  I had to break out each individual P2P rule as a line item in the firewall.  Pain in the A$$!  Plus if a new category gets added I won't be protected unless I check on it periodically. Just one more thing to do.  C'mon Meraki we need to allow an exception.

[Dunky]

We need to be able to have an ALLOW and also src and dst fields.

It's so annoying when internal inter-vlan traffic is getting blocked by L7 NBAR.

Meraki just haven't thought this through !!

[Erminio]

We have reverted back to 15.44 for approximately 70 MX's. Disabling the NBAR ID database didn't work (Under Network/General/Traffic Analysis change to "Basic: collect generic traffic categories". Under the surface it is still hitting the NBAR ID database with "false positive" while it is disabled. We are waiting for a fix to this NBAR ID problem.

[HScar]

I guess I'll pile on the list here.

Awesome NBAR is blocking Google Chrome Sync on TCP 5228. Thinks it's Google Hangouts/Streaming Video Category on Layer 7. NBAR 1087.

Also blocking OpenDNS under NBAR ID 1087. UDP 53 and thinks it's Google Hangouts too. There's another thread on this.

Who designed this garbage?

[MarcAEC]

Let's not forget that this problem is really an example of a deeper problem.  The Merkai development team likes to design functionality that has broad effects without including a way for customers to implement exceptions.  The NBAR problem would be much smaller if there was a way to create ALLOW rules for confirmed mis-identifications.  Instead, it's an all or nothing choice.  This is the same problem with the geographic filtering.  Need to access one specific site in Russia?  Guess what --you need to allow all inbound and outbound traffic from the entire country. 

[HScar]

Totally agree. I'm so disappointed in their current design in L7/NBAR Firewall.

Just a side note, I have had some success using FQDN's in Layer 3 to bypass country blocks.

[MarcAEC]

Had Encrypted P2P / Statistical Peer-To-Peer start blocking WMI traffic between severs today.  At least we know now to check the Meraki event and security logs first when when things start randomly failing,

[MarcAEC]

When I originally set up the Layer 7 rules, I thought it would apply to internet traffic only, not traffic between clients and servers within the local network.  Has it always been like this, or is application to internal traffic something new-ish?  IMO, this makes the lack of the ability to add layer 7 allow rules even worse.

[Dunky]

To the best of my knowledge its always been like that.

Until such site as they give us the ability to add ALLOW rules and src/dst IP's its unusable!

I have had to permit Statistical Peer to Peer which applies to ALL traffic purely to allow management of an internal phone system!

[etb]

not that I want to weaken our case for improved NBAR and related functions 😀, but if you have some firewall rules that you don't want to apply at all in some places, you could probably use group policies.

[Dunky]

I guess I could put specific clients into a Group Policy, however that does not address the underlying issue of the L7 rule applying to ALL traffic as we cannot specify src/dst IP's.   So what if its Stastistical Peer-to-Peer if it's between internal IP's? I dont care, but we have no way of locking this down so even with a separate Group Policy applied that would still allow Statistical P2P on ALL traffic, not just between my internal endpoints.

[happyclients]

Has anyone else had any issues with NBAR blocking statistical p2p?  Dunky I just PM'd you to get the case number for your issue, thank you in advance.  We're seeing the same problem where our in house application traffic is being NBAR classified as statistical p2p with no way to whitelist.

[jrsilvius]

Ongoing issue. We've had issues with it since they introduced NBAR. They keep telling us they are going to fix it, but we've tried all the firmware updates currently available and still have the problem. Sadly, the only way to fix it that I've found so far is to allow the P2P category of your application and hope that nothing malicious takes advantage of it.

[thunt]

Agreed.  Sad.   We have battled (and lost) the same issue.

[happyclients]

Ok thanks folks.  This is unfortunate.  So in order to allow our app that's being classified as "statistical p2p", how exactly would that be accomplished?   It doesn't look to be being blocked by an explicit rule it lists the L7 firewall rule as "Deny" which is odd.  That sounds like more of the "action" as opposed to a rule name. 

[MarcAEC]

In your Layer 7 Firewall Rules make sure the following is not present:

Deny # Peer-to-peer # All peer-to-peer

Deny # Peer-to-peer # Encrypted P2P

I've also had to remove:

Deny # Peer-to-peer # Encrypted P2P # eDonkey

Deny # Peer-to-peer # Encrypted P2P # eDonkey Static

Deny # Peer-to-peer # Encrypted P2P # Encrypted eMule (dDonkey & Kademlia)

[Isack2230]

Hey everyone -- I opened a new ticket on slightly related topic as we were seeing blocks due to NBAR ID: 1889 but no classification back in early decemember.  Here was Supports response:

Is there a more recent event of this happening? The latest Event Log entry for this client as a Layer 7 firewall rule is on Dec 2. We actually removed NBAR 1889 from our ruleset on Dec 13th so you shouldn't see this hit any longer.

Regards,
Bryon Adams 

I'm not sure if that addresses some of these issues we've all put workarounds in for but I wanted to share

[etb]

Thanks Isack2230!

Looking at our Traffic Analytics page, I can see that the traffic which used to get classified as Statistical P2P maybe 50% of the time now appears to be getting classified as "Unknown".  I actually don't have any Statistical P2P showing up in the past 30 days, although most users use this application every day.

In addition to "Unknown", I also have 2 separate rows for "undefined" which I don't recall seeing before.  Those 2 "undefined" rows combined would be the 3rd largest single traffic type in our network over the past 30 days, so I'm guessing that they're just some kind of a catch-all for traffic for which NBAR doesn't have any definition.  Though I don't know how that differs from the traffic which gets classified as "Unknown".  If I click on "undefined" to try to view the clients and hosts, the screen is just blank.

But, at least it looks like our legitimate traffic won't get blocked as P2P anymore.  When I'm feeling adventurous, I'll try re-enabling the P2P content filtering and L7 firewall rules.

[AlexP]

The above post from @Isack2230  is correct - after some discussions with the folks who maintain the NBAR protocol packs, we got some clarification from them that the "statistical" rules are all considered "best-effort" on their part. Per some requests, the protocol pack documentation has been updated to ensure it's more clear these rules may not be as accurate as others: https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/qos_nbar/prot_lib/config_library/nbar2-protocols/n...

For now, because of the continued issues we've seen reported with its accuracy and false positive rate, the rule has been disabled on Dashboard, and should no longer be configured or configurable.

Side note: it remains to be seen if complaints come up with any of the other "statistical" rules in the protocol pack, since they're all similarly flagged as potentially being inaccurate in the PPACK documentation. We aren't currently entertaining ideas to disable any other rules, but we'll be keeping an eye on complaints and case data that might prompt us to reconsider this position in the future.

[PhilipDAth]

I'm kinda thinking only statistical models with an accuracy of at least 99.9% should be considered when it comes to blocking (as opposed to just monitoring).

Even at 99.9% accuracy, 1 in a thousand applications will be misclassified and incorrectly blocked.

The NBAR team really needs to also publish the confidence level for these models.  Does "best effort" mean the machine learning model was 68% confident, 95% confident, what?

[AlexP]

---

@PhilipDAth wrote:

I'm kinda thinking only statistical models with an accuracy of at least 99.9% should be considered when it comes to blocking (as opposed to just monitoring).

Even at 99.9% accuracy, 1 in a thousand applications will be misclassified and incorrectly blocked.

The NBAR team really needs to also publish the confidence level for these models.  Does "best effort" mean the machine learning model was 68% confident, 95% confident, what?

---

This is great feedback - I will bring definitely bring this up and see what additional context we can get.

[Dunky]

@AlexP Are there plans to allow us to specify permit/deny and src/dst for the L7 rules?  Many a time I want to permit for internal apps but I cant so have to remove the L7 rule for internal apps to work which means its enabled for all traffic, not just between the specific src/dst that I need to permit it for.

[AlexP]

I cannot comment on feature developments in that regard right now unfortunately.

[Midway1]

Any further updates on NBAR 1889 issues? I am still canceling scheduled upgrades beyond 15.44. Any word if the "18.X.X" firmware allows rules for L7 to allow Stat P2P?

[AlexP]

The ability to block statistical P2P (NBAR 1889) as a category was disabled before the Holidays in 2022 as I noted in this response: https://community.meraki.com/t5/Security-SD-WAN/MX-NBAR-blocking-Statistical-Peer-to-peer-traffi/m-p...

This includes removing it from all aggregate or "All X" rules it may have been a part of