AnyConnect Open Beta 16.4 Broke Office365 and Other SaaS Apps

[41D5]

I have several sites working in the AnyConnect closed Beta for a long time.  Probably almost a year.  I was excited to see AnyConnect move to public beta so I started scheduling 16.4 upgrades.  We didn't make any changes before or right after the 16.4 upgrades.  We did post upgrade testing and everything seemed fine.  However, after we moved one site to 16.4 users started complaining that their Outlook showed as disconnected (Using Office365 with a hybrid AD setup).  

The problem manifests itself about 80% of the time.  Sometimes it works fine, but we generally don't have a problem reproducing it.

Here is our environment:

2 MX250's in HA w/Virtual IP (16.4)

2 ISP's

Windows 10 laptops

Office 365 Hybrid AD Setup

AnyConnect Client v4.9.06037 for Windows

Full Tunnel Configuration

AnyConnect / Internal users / IPSEC Client VPN all use the same DNS servers

From our testing we found:

Clients connected to the AnyConnect VPN on our Meraki MS250's are getting disconnected from Office365 and other web (SaaS) applications.

• If the same client connects to the IPSEC Client VPN we do not see the problem
• If the same client works from inside the office we do not see the problem
• If the same client connects to our legacy ASA5520 using the same exact AnyConnect version on the same computer we do not see the problem
• If the same disconnects from the VPN and uses Office 365 from their home Internet we do not see the problem

We took took the following steps and retested the scenarios above:

• Disabled AMP/IDP - no change, AnyConnect still broken
• Removed country blocks - no change, AnyConnect still broken
• Permit Any on the top of the firewall rules (UGGH, but had to rule it out) - no change, AnyConnect still broken
• Verified no weird traffic shaping stuff - no change, AnyConnect still broken
• Forced all traffic to ISP1 - no change, AnyConnect still broken
• Forced all traffic to ISP2 - no change, AnyConnect still broken

So we opened a case with Meraki support.  I received the quickest response on a case that has ever happened: "Please upgrade your MX to 16.5 and see if it fixes the issue."  Well the release notes for 16.5 weren't much help but since we were out of options we upgraded to 16.5 and...... - no change, AnyConnect still broken  😥

I want my closed Beta code back!

Anyone else experiencing an issue like this?

I am going to call support back so we can take captures and perform other science experiments while the client happily uses their legacy ASA5520 AnyConnect VPN that works fine.  Sigh....

I want my AnyConnect closed beta code back!

[PhilipDAth]

No issues here.

I'm going to guess it is something to do with the tunnelling and firewall rules blocking it.  Are you running full tunnel or split tunnel?

Second guess - have you got internal DNS overriding some external DNS settings?

What do Office 365 diagnostics say?

http://diagnostics.office.com/ 

[41D5]

@PhilipDAth ,

I think these suggestions are great.  We have a full tunnel solution so I added that to my original post as that is an important design consideration.  I also validated that all of our VPN solutions and internal clients are using the same DNS.  

I actually never used the Office365 diagnostic tool.  It was pretty cool so thanks for the tip!

I will update this thread as we get more info and establish root cause.

[akh223 · Accepted solution]

Not sure if this will help, but I have identified and opened a case with Meraki Support about our MX's that we are testing 16.4 on.  16.4 is incorrectly identifying our SAP application traffic as P2P application (bittorrent), and since I have a rule that blocks all P2P applications it is getting blocked.  I wonder if some of your O365 traffic is being classified as an incorrect application and being blocked by the Layer 7 firewall rules?

Do you have any Layer 7 firewall rules?  If so, can you disable them and see if everything starts working?

If you are dumping all the MX logs to a syslog server, you can search the syslog messages for "l7_firewall" to see if you are getting traffic blocked.

I dump ours in to splunk, and it shows up like this:

May 5 21:46:25 10.x.x.x 1 1620265585.486264487 XXX_FW01 l7_firewall src=10.X.X.X dst=X.X.X.X protocol=tcp sport=58700 dport=3299 decision=blocked

Hope it helps.

[41D5]

@akh223 ,

I could hug you!  YES!!!  Once I saw your post I knew this had to be it.  We do have a L7 FW rule for P2P traffic.  I disabled that rule and it seemed to fix the issues and the other issues haven't presented themselves.  I am 99% sure this fixed it based on testing and given a day or so with no issues I think it will be 100%.  I will also update my Meraki case to see if I can get this logged as an official bug.

THANK YOU, THANK YOU!

[akh223]

Glad to help!!  On firmware version 15.42 our SAP traffic works properly, but we found another application that was incorrectly categorized as bittorrent.  Of course, you cant do the AnyConnect VPN stuff on 15.42, so you are at the mercy of Meraki support and the next version of code they release.

[AnythingHosted]

16.6 is now out with some AnyConnect fixes

[akh223]

Not sure about the AnyConnect fixes, but 16.6 is still incorrectly categorizing SAP client traffic on my test box.