Office365 issues after connecting MX64

[CammackIT]

Hi all -

New Meraki user here. We just migrated from a Cisco ASA5510 to a MX64 unit. All the firewall configurations went fairly smoothly, the only issue we're seeing is that our connections to Office365 are inconsistent and cause Outlook to hang very frequently. I've tried traffic shaping rules on both default and off, and included a shaping rule setting all email to high priority. I've specifically whitelisted office.com, office365.com and outlook.com and also added flow preferences for all the Office365 IPs. All of the above action doesn't seem to have any effect, and our email is at a standstill. Has anyone run into this before, or have any recommendations? Thanks in advance for your help!

Sean

[BlakeRichardson]

@CammackIT  are you using O365 outlook via a web browser or email client?  

Have you looked at the event log to see if its reporting any errors?

[CammackIT]

Most users are connecting via Outlook2010. I have checked event logs, and the only thing I get is a generic AppHang - Outlook has lost connection to Microsoft, followed by a Windows Error log:

Source:        Windows Error Reporting​

Date:          11/13/2019 4:36:47 PM​

Event ID:      1001​

Task Category: None​

Level:         Information​

Keywords:      Classic​

User:          N/A​

Computer:      ***************************​

Description:​

Fault bucket 1251049214596579294, type 5​

Event Name: AppHangB1​

Response: Not available​

Cab Id: 0​

Problem signature:​

P1: OUTLOOK.EXE​

P2: 14.0.7230.5000​

P3: 5c673b81​

P4: 87fc​

P5: 134217728​

Nothing specific to Outlook in the application logs either. 

[NolanHerring]

Is your Internet connection maxing out by any chance? One thing I know and hate about OFFICE 365 is that it is a bandwidth hog on a huge scale. It's a known thing and its due to the way it is constantly syncing the client to the cloud etc. It really drives me nuts actually and I want to punch a wall.

That being said, I did a quick google search on the error you posted above and its coming back with a TON of other people with similar issues, which most are not pointing to a firewall issue but more of an application issue (possibly with addons).

[BlakeRichardson]

@CammackIT  sorry I meant the Meraki event log not the Windows one. 

[CammackIT]

These are the only entries in the event log, although they repeat every 30 minutes.

Nov 14 07:29:40	70:71:bc:53:39:a2	Source IP and/or VLAN mismatch	source_client_ip: 192.168.0.71, source_client_mac: 70:71:BC:53:39:A2, source_client_assigned_vlan: 1  « hide last_illegal_ip 10.1.10.3 client_total_illegal_packets 24027 all_total_illegal_packets 52127 last_reported_total 51450
Nov 14 06:59:40	00:50:56:ae:00:01	Source IP and/or VLAN mismatch	source_client_ip: 169.254.229.95, source_client_mac: 00:50:56:AE:00:01, source_client_assigned_vlan: 1  « hide client_total_illegal_packets 4165 all_total_illegal_packets 51450 last_reported_total 50787

[CammackIT]

It's not. In fact, our throughput has increased about 30-50x with the new MX appliance. The average DL/UL speed with the ASA5510 was 3-5Mbps, and the MX is in the 170Mbps range. 

As to the application specific issue, while I'm open to that possibility, when I switch back to the ASA5510 as the front edge device, the O365 issues disappear. There are no specific rules on the ASA5510 relative to O365/Microsoft.com etc. either.

[HitoshiH]

@CammackIT 

Has the MX not been configured with full tunnel to VPN hub which could have been adding additional latency to Internet traffic including Saas application (Office365) access?

If it had been configured as full tunnel, changing it to split tunnel (Untick of default route to VPN hub) would reduce latency much.

In addition, setting up traffic shaping for Productivity > Windows Office 365 as high priority and shape non-business critical application such as streaming Youtube, gaming etc would give priority for business critical application and Office365 application.

If stability does not stay even the above is configured and it has been configured as split tunnel (Direct internet access), isolating issue if it was cased by high latency or delay response of HTTP(S) to application would be an idea to troubleshoot it. (Meraki Insight provides more visibility and troubleshooting for Saas application including Office365 though)

Hope this would help some! 

[CammackIT]

The MX is solely acting as the front edge Layer 3/7 device, no VPNs are configured. All clients experiencing the O365 issues are internal.

I do have traffic shaping rules in place delegating O365 (and all email functions) as high priority. I have internal controls on access to YouTube/gaming/etc in place so they are not relevant.

I'll check out the Meraki Insight tool - if I can find it! 😉 Thanks for the advice.

[Uberseehandel]

I have a recent version of Win 10 Pro (1809 OS build 17763.864), BitDefender Total Security with default browser set to Edge Beta Version 79.0.309.18 (Official build) beta (64-bit). All the Office365 applications are 32/64-bit Office2016, kept current.

Currently, I am at Institute A, a world leading Health Research organisation, with effing awful IT infrastructure, out of date and insecure software and a really bad attitude towards addressing these issues. My laptop is connected to their poorly set up WiFi, but I'm not unduly worried as Office365 sets up an encrypted link to the remote services. Until it all goes titsup (technical engineering term, much used by Grace Hopper). BitDefender and Office apps throw a wobbly to do with Azure/Exchange/Office cert inconsistencies, many web sites become unreachable (including Meraki). What brings this on - setting the laptop to sleep; on waking all hell breaks out. Fixing it required restarting, forgetting the WiFi site, doing an offline Office365 repair and briefly switching to Chrome. This took a very long time to fix.

Next door is Institute B, another world leading Health Research organisation. Where i am based, I have the choice of connecting to either Institute's WiFi. I never have these issues with Institute B.

So, in my case, a problem that sounds very like yours is triggered by really crap network infrastructure and software. I'm not implying anything about other people's systems.

Hopefully, my experience has given you some clues.

Note expletives have been deleted; there may be some remaining technical terms that somewhat resemble expletives, but by general accord and usage they terms commonly encountered in network engineering.

[CammackIT]

If nothing else, your post made me laugh. 😄 

I work for a non-profit that thinks IT ranks behind cutting down trees in the parking lot. So, yes, I have crappy outdated infrastructure which has certainly caused many headaches. (Until last year, they were running on Exchange2003, on an WinServer2003 box...) It is certainly a possibility that improving one aspect of security can make an older system fail, my thought is the huge increase in our bandwidth. I'm trying to throttle email back down to ASA5510 levels to see if that helps (why it would, I don't know, but I'm grasping at straws). Thanks for pitching in!

[Prashant_India]

@CammackIT  please use FQDN feature of meraki for O365 application.

[PhilipDAth]

>Outlook2010

This may be your problem.  Microsoft dropped support for Outlook 2010.

https://www.corbel.co.uk/microsoft-ends-support-office-365-outlook-2007-2010-clients/ 

"Microsoft won’t deliberately prevent you from connecting to the service but the quality of your Office 365 experience will diminish over time."

[CammackIT]

I am aware of the deprecated support for Office 2010 (so is our board, as I tell them every month). However, if I put the ASA5510 back in place as the front edge device, the issues go away. They are only present with the MX device.

[HitoshiH]

@CammackIT 

I have one more point that I would check if TCP retransmission is not occurring due to mismatch of MTU.

The default MTU on MX is 1500 byte which can be adjusted by Technical support through support case (Help > Cases on dashboard).

If you take packet capture on Internet interface where traffic to office365 is and saw TCP retransmission, or you have MTU value other than 1500 byte configured on WAN interface on ASA, MTU mismatch could be reason making TCP communication so slow due to retransmission.

Hope this would help!

[Uberseehandel]

After a titsup event (see my post above), my inability to access certain websites using a browser (Edge Beta) was reminiscent of the behaviour I experienced whilst I was trying to optimise/maximise packet sizes to match the Jumbo capability of the boards in the FTTC cabinet.

However, I did not have to adjust any of the packet size parameters to solve this problem, just repair the Office installation (offline) and forget the crapulous network's WiFi SSID. Personally, I don't think there was anything wrong with the Office installation, but possibly once MS has flagged the installation as faulty, one has to clear the flag.

It would be worth checking what the maximum packet size that the ISP supplied equipment/modem will pass is; PPPoE will add bytes to the 1500 the MX will pass.

Memo to Meraki - in a Fibre World, we should be able to pass 9K packets 🤓

[CammackIT]

The ASA5510 MTU was also 1500 so I don't see that as the issue. Thanks for the suggestion though!

[BlakeRichardson]

@CammackIT  I would disable all traffic shaping rules and put everything back to default and see if the problem still persists. 

[PhilipDAth]

Do you perhaps have an "sysopt connection tcpmss ..." line on the ASA?

What type of "outside" connection is it?   PPPoE, static IP, etc?

[CammackIT]

No "sysop..." line, and both inside and outside interface is/was static IP. 

[PhilipDAth]

How many users have you got?

[CammackIT]

10-12 users that use email/O365 regularly. There isn't a heavy workload that would explain any significant lag.

[CammackIT]

As far as traffic shaping goes, I've:

1) disabled all rules,

2) ran with just the default set, 

3) ran with just the Office365 (in Productivity) set on high priority,

4) set all email to high priority and

5) basically every combination of 2-4 (2+3, 2+4, 3+4, 2+3+4) 

and the issue persists. So.... 😞

[HitoshiH]

@CammackIT 

Thanks for the confirmation of MTU value on the interface.

How about uplink configuration (Security & SD-WAN > SD-WAN & traffic shaping > Uplink configuration > WAN1 / WAN2) if those interfaces bandwidth have been configured properly or not? (It is recommended to match the bandwidth which ISP provides to the MX80)

This is to avoid policing on Provider Edge router drops packets sent from MX.

Hope this would help!

[Loquanet]

I see nobody helped you solve the issue. 

If thus far didn't work why not buy a VPN service and bypass all DPI and shaping. Just as a test and maybe a permanent access to internet. 

I can suggest 2 (of the best this far) vpn providers. 

[Uberseehandel]

@Loquanet 

Spam Spam Spam

https://www.youtube.com/watch?v=mBcY3W5WgNU

First post - selling something - easy to spot.

[Loquanet]

@Uberseehandel
Not at all spam! Just when I avoided to plug any vpns service to use which would have been spammy...a misguided ignorant like you quoted a YouTube like that's a definite expert on spam or can read minds like a God. I didn't sell anything and didn't want suggest any service of vpn.

[Uberseehandel]

@Loquanet 

I wasn't born yesterday.

Was that your standard reply because the reference I posted went straight over your head.