Meraki MX64, MX67, MX100 issues with 16.16 Audio/Video issues Team, Youtube etc...

[Tim_Patrick_ADS]

We have a small handful of sites on the 16.16 code and we had some reports of performance issues, horrible team audio quality, videos becoming choppy, digital signage not playing audio and video in sync, as well as latency issues for some client-server manufacturing applications hosted between locations.

We went down the usual suspects looking for congestion, internet circuit issues etc..

While looking at the newest site still coming online I noticed nothing was showing in the traffic analytics, this makes me wonder if NBAR or something else is causing issues with packet classification and QoS . We have just downgraded to 15.44 I will post an update in the AM when I hear back from my users 

[GIdenJoe]

I have no customers yet complaining about this.
I did have a colleague complaining that a certain MX64 doing interVLAN routing only having a throughput of about 30 Mbps but I'm not sure if they were running that firmware version.

It would be handy to see the status of the Queues and to see what packets are put into which queues in case of congestion.

[Jordan1]

I'm curious if the downgrade resolved the issue? We are looking at upgrading a number of sites to 16.16 but would obviously like to avoid these type of issues. Did you happen to upgrade to 16.16.1 or just 16.16? 

[Tim_Patrick_ADS]

My users have been non communicative if this helped, I have migrated two more sites to 16.16 from 15.44 this weekend to see if they have the same performance issues so far that have not so it could have been a strange anomaly at that site. 

[Jordan1]

Okay thanks, I know how that goes. Here's to hoping it was just an anomaly! 

[Tim_Patrick_ADS]

I set up a test lab and am able to replicate the issue 16.16 causes bad SD-WAN latency and packet loss rolling back to 15.44 and the problem resolves. Pushing on TAC

[Tim_Patrick_ADS]

I created a new test network since I am not getting reliable data form my user. I have an MX64 running 16.16 when in MS teams the audio quality was bad it would cut out freeze kick up poor audio quality messages over and over. I also did a continues ping to a server in my data center (not the real IP bellow). You can see the results at 16.16 then the window of the MX rebooting to roll back to 15.44 and the change in latency and packet loss.

I have a case open with TACX but wanted to bring this to other peoples attention.

Reply from 10.10.10.10: bytes=32 time=78ms TTL=126

Reply from 10.10.10.10: bytes=32 time=86ms TTL=126

Reply from 10.10.10.10: bytes=32 time=74ms TTL=126

Reply from 10.10.10.10: bytes=32 time=74ms TTL=126

Reply from 10.10.10.10: bytes=32 time=84ms TTL=126

Reply from 10.10.10.10: bytes=32 time=71ms TTL=126

Request timed out.

Reply from 10.10.10.10: bytes=32 time=165ms TTL=126

Reply from 10.10.10.10: bytes=32 time=80ms TTL=126

Reply from 10.10.10.10: bytes=32 time=136ms TTL=126

Reply from 10.10.10.10: bytes=32 time=137ms TTL=126

Reply from 10.10.10.10: bytes=32 time=67ms TTL=126

Reply from 10.10.10.10: bytes=32 time=104ms TTL=126

Reply from 10.10.10.10: bytes=32 time=135ms TTL=126

Reply from 10.10.10.10: bytes=32 time=75ms TTL=126

Reply from 10.10.10.10: bytes=32 time=96ms TTL=126

Request timed out.

Reply from 10.10.10.10: bytes=32 time=146ms TTL=126

Reply from 10.10.10.10: bytes=32 time=75ms TTL=126

Reply from 10.10.10.10: bytes=32 time=89ms TTL=126

Reply from 10.10.10.10: bytes=32 time=74ms TTL=126

Reply from 10.10.10.10: bytes=32 time=141ms TTL=126

Reply from 10.10.10.10: bytes=32 time=80ms TTL=126

Reply from 10.10.10.10: bytes=32 time=109ms TTL=126

Reply from 10.10.10.10: bytes=32 time=77ms TTL=126

Reply from 10.10.10.10: bytes=32 time=79ms TTL=126

Reply from 10.10.10.10: bytes=32 time=79ms TTL=126

Reply from 10.10.10.10: bytes=32 time=137ms TTL=126

Reply from 10.10.10.10: bytes=32 time=95ms TTL=126

Reply from 10.10.10.10: bytes=32 time=85ms TTL=126

Reply from 10.10.10.10: bytes=32 time=79ms TTL=126

Reply from 10.10.10.10: bytes=32 time=98ms TTL=126

Reply from 10.10.10.10: bytes=32 time=81ms TTL=126

Reply from 10.10.10.10: bytes=32 time=84ms TTL=126

Reply from 10.10.10.10: bytes=32 time=77ms TTL=126

Reply from 10.10.10.10: bytes=32 time=69ms TTL=126

Request timed out.

Reply from 10.10.10.10: bytes=32 time=75ms TTL=126

Reply from 10.10.10.10: bytes=32 time=71ms TTL=126

Reply from 10.10.10.10: bytes=32 time=78ms TTL=126

Request timed out.

Reply from 10.10.10.10: bytes=32 time=226ms TTL=126

Reply from 10.10.10.10: bytes=32 time=67ms TTL=126

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Reply from 10.10.10.10: bytes=32 time=22ms TTL=126

Reply from 10.10.10.10: bytes=32 time=23ms TTL=126

Reply from 10.10.10.10: bytes=32 time=22ms TTL=126

Reply from 10.10.10.10: bytes=32 time=59ms TTL=126

Reply from 10.10.10.10: bytes=32 time=24ms TTL=126

Reply from 10.10.10.10: bytes=32 time=23ms TTL=126

Reply from 10.10.10.10: bytes=32 time=22ms TTL=126

Reply from 10.10.10.10: bytes=32 time=22ms TTL=126

Reply from 10.10.10.10: bytes=32 time=23ms TTL=126

Reply from 10.10.10.10: bytes=32 time=24ms TTL=126

Reply from 10.10.10.10: bytes=32 time=23ms TTL=126

Reply from 10.10.10.10: bytes=32 time=22ms TTL=126

Reply from 10.10.10.10: bytes=32 time=23ms TTL=126

Reply from 10.10.10.10: bytes=32 time=86ms TTL=126

Reply from 10.10.10.10: bytes=32 time=22ms TTL=126

Reply from 10.10.10.10: bytes=32 time=23ms TTL=126

Reply from 10.10.10.10: bytes=32 time=24ms TTL=126

Reply from 10.10.10.10: bytes=32 time=23ms TTL=126

Reply from 10.10.10.10: bytes=32 time=24ms TTL=126

Reply from 10.10.10.10: bytes=32 time=27ms TTL=126

[Jordan1]

Thanks for posting this. I am curious what your test environment looks like? Specifically are you running the Teams traffic through the VPN (full tunnel)? Also, what do your layer 7 rules look like? I'm not blocking much using the layer 7 rules so I'm wondering if maybe I wouldn't be affected as much moving to 16.16 or 16.16.1.