Latency and Slow Internet Speeds

[TechB0SS]

I have a full Meraki stack in a higher-ed environment, (2) MX 450 security appliance, (100+) MS 250 switches, and (500+) MR 33 42 and 84 wireless access points. Over the past few days, i have been complaints from users about latency and slow Internet speeds. My ISP connection is 5GB, split between both MX appliances. Daily bandwidth consumption is average 1.5GB, so not reaching our limits there. On a daily average we have about 5,000 clients connected. There is (1) traffic shaping rule on the public wireless to limit clients to 20Mbps, but when running a speedtest, clients are averaging 1.2Mbps.

I have been looking in my dashboard and organization summary reports trying to figure out what is going on, and noticed that my Public Safety department's non-Meraki IP cameras and NVR's have high usage, one NVR usage is 1.04TB daily. Most of the cameras are 80+ GB usage. The MX appliance device utilization is 90-100 percent daily. I'm trying to find the smoking gun, but nothing obvious jumping out. Could it be the security cameras and NVR devices causing latency and slow Internet speeds for everyone? How should I go about controlling usage of these devices?

Not sure what else could be causing these slow internet speeds...

[BlakeRichardson]

@TechB0SS  Without knowing how your network is laid out its very hard ot say. Yes traditional IP camera to NVR setup's move a lot of data. 2/3 on my networks data is CCTV cameras. 

Have you tried testing the internal network? i.e. is there any latency to something like your internal DNS server?

Do you notice any hugely noticable improvement between wireless and wired client deivces?  

[GreenMan]

A few initial thoughts, from reading your post:

1)  Average utilisation over (say) a day can often be misleading.   It's no good your infrastructure carrying next to nothing between 4PM and 8AM, if it's totally swamped for the rest of the time - but if the stats you look at are only averaged over the day, you very likely won't see the problem.   You need to look at more granular stats during the periods that people are having the issue.

2) 90 - 100% utilisation of an MX for any length of time is likely to cause problems for many of the applications flowing through that MX.   Maybe look at waht traffic is required to run through the MX - asssuming your inter-VLAN traffic is all L3 routed on the MX, what are the major traffic contributors, application and client-wise?   Could they, for instance, be located in the same VLAN, without overly damaging your security?

3) In regard to 3, I would certainly look at what your cameras & NVR etc. are communicating with, and how that would be handled by your LAN setup

If you haven't already, don't be afraid to ask for help & advice from Meraki Support with this

[PhilipDAth]

Are both wired and WiFi users reporting the issue, or just WiFi?  If WiFi, check out the WiFi Health tool.

https://documentation.meraki.com/General_Administration/Cross-Platform_Content/Meraki_Health_Overvie... 

Is it a specific Web app (like Office 365) or "everything" having the issue?  If it is a specific app, Meraki Insight is helpful - but this is something extra you have to pay for.

https://meraki.cisco.com/products/meraki-insight/ 

Have any changes happened in your environment around the time users start complaining?  Firmware updates?  Server upgrades?  Microsoft patch on workstations?

[TechB0SS]

I think I found the fix? Sounds too simple, but a reboot of the MX450. Since the reboot on Friday, approximately 48 hours ago, device utilization is about 15% with 5800 clients connected.  Keeping an eye on it…