Meraki MX QoS

[newengineerhere]

Hello, I'm hoping to get some answers on the way Meraki MX QoS works as I couldn't find the information I needed in the documentation.

The background information is that we have a number of branch sites with an MX connecting back to the hub site (H), where there is a MX250 in One-Armed VPN Concentrator Mode. I want to understand what the options are for configuring traffic-shaping at all sites and what exactly those configurations will accomplish.  I am interested in the settings under Security & SD-WAN > Configure > SD-WAN > traffic shaping > all the settings starting at “Traffic shaping rules.” 

1. Regarding the “Priority” setting, the documentation states the following:

Specifying a traffic shaping rule as High, Normal, Low guarantees a certain fraction of the uplink to each priority level. The ratios are as follows:

• High       4/7
• Normal   2/7
• Low       1/7

Other than allocating the above fractions of the overall bandwidth to the three different traffic queues, does this setting have any other effect on how the traffic is treated?  In other words, 4/7 of the overall bandwidth is allocated to High and 2/7 to Normal, but are packets otherwise treated any differently based on whether they are assigned to the High vs Normal queue? 

2. If the traffic assigned to the High queue has reached the available limit, will all additional traffic be dropped in all scenarios, or is it possible that additional traffic will be forwarded if the Normal and/or Low queues are not currently forwarding their respective maximum amounts of traffic?

3. I ask the above questions because my guess is that the packets are not treated any differently based on the queue they are assigned to, and this setting simply allows me to allocate more bandwidth to traffic that I deem more important, but if that’s true, then referring to the setting as “Priority” is misleading.  If all queues are treated equally, then this setting does nothing other than choose which traffic should be assigned to queues of different sizes.  Is this correct?

4. I assume that the “DSCP tagging” setting does nothing other than mark packets with DSCP tags based on whether or not they match the rule definition.  Is this correct?

5. The purpose of DSCP tags is to provide a device with instructions on how to treat traffic classes differently in terms of precedence and drop probability.  What do MX appliances actually do as far as processing traffic based on their DSCP tags in terms of precedence and drop probability?  Please include as much detail as possible regarding how packets are assigned to different queues and how the MX decides which packets should be dropped and when they should be dropped.

6. If the answer to #5 is that the MX does not process traffic differently based on DSCP tags, does that mean that the MX is limited to simply marking packets with DSCP tags (as opposed to queuing/shaping/policing them differently)?

7. If the answer to #5 is that the MX does process traffic differently based on DSCP tags, how does the MX treat that packet differently compared to a packet with a different DSCP tag?  Every platform uses its own definitions on how to treat different DSCP tags, but I haven’t seen any documentation on how Meraki devices do so.

8. If the answer to #5 is that the MX does process traffic differently based on DSCP tags, then when the MX receives a packet without a DSCP tag, but it is configured to tag that particular packet with a specific DSCP value, how does the MX treat that packet differently compared to a packet with a different DSCP tag?  In other words, if the MX does treat traffic differently based on the DSCP tag, does it only do so if it receives a packet with a DSCP tag, or does it also do so if it receives a packet without a DSCP tag but has a rule to tag that packet with a DSCP tag?

9. All of our branch sites use Internet connections (as opposed to MPLS, for example), so once VPN traffic leaves the MX it is encrypted.  I would assume any DSCP tagging on traffic leaving a branch MX would not be visible to any upstream devices until the traffic reaches the hub MX.  Is this correct?

10. We are planning to reconfigure the VPN routing so that the default route at branch sites will be sent directly out to the Internet rather than over the VPN back to the hub.  If the answer to #5 is that the MX does process traffic differently based on DSCP tags, does the answer to #5 depend on whether that traffic is sent over the VPN versus directly out to the Internet?

[NolanHerring]

Tagging some others on here since this is a great post and I'm sure will provide tons of info for others:

@PhilipDAth 

@MerakiDave 

@BrechtSchamp 

I'll try my best at this, so please excuse any errors on my behalf.

1. Fraction of uplink is based on percent, meaning if you have 100Mbps circuit, you 'HAVE' to configure the uplink as 100 (you can't leave the default of say 500 or whatever it is, the max). It uses the configuration you set to ensure that the percent of bandwidth is available for the queue slots. My interpretation is that if LOW is 1/7 and its pushing more, it will essentially just drop it (if the rest is already being used), but if not then it will allow it to grow.

2. I believe it will expand if the other queue are not being used, and if all queues are full then it starts killing off LOW first, then Normal etc. At least that makes the most sense to me.

3. I yield the remainder of my time to my opponents =P

4. Correct, they will either honor or remark them as you have configured. Keep in mind Internet does not care what the marking is since you can't QoS Internet traffic.

5. Hmm...probably yielding more time to others but my best guess was it would apply for only AutoVPN destined traffic. So voice packets (EF) having higher priority over Background etc., when getting processed in the queue.

6 through 9: have a read here:

https://community.meraki.com/t5/Security-SD-WAN/QoS-over-VPN-tunnel/td-p/33078

https://documentation.meraki.com/MX/Firewall_and_Traffic_Shaping/Using_Packet_Prioritization_on_a_Tr...

10. You mean your going to enable split-tunneling?

[PhilipDAth]

This is really 10 questions.  Rather than one massive reply, I'll bite it off in several.

>1. Regarding the “Priority” setting, the documentation states the following ...

>2.  If the traffic assigned to the High queue has reached the available limit ...

>3. I ask the above questions because my guess is that the packets are not treated any differently...

Lets say you have 100 packets waiting in the queue to be sent, using all different priorities.

The first 4 "high" priority packets are sent.  Then the first "2" normal priority packets are sent.  And finally a single "low" priority packet is sent.  Repeat until the queue is drained.

Lets say you go to send 3 high priority packets but there is only 1 waiting.  That 1 packet gets sent and then it moves straight onto sending the next highest priority traffic.

If there is no high priority traffic then 2 normal priority packets get sent and then 1 low priority packet gets sent.  Effectively normal priority packets not get 2/3 of the link and low priority now gets 1/3.

There is one special except to all of this.  Packets with a DSCP marking of EF (which is normally only use for VoIP RTP packets) always get sent first in the priority queue.  So if their was 100 EF marked packets in the queue they would all be sent before anything else.

Typically packets wont get dumped.  However there is a limited amount of buffering (I don't know that number).  So if you offer 10Mb/s of load to a 1Mb/s circuit (so 10 times as many packets get queued up as what can be sent) then obviously you will eventually exhaust buffering and the excess packets will get dumped.

Apart from EF DSCP marked packets, all packets are treated equally.

[whistleblower]

---

@PhilipDAth wrote:

There is one special except to all of this.  Packets with a DSCP marking of EF (which is normally only use for VoIP RTP packets) always get sent first in the priority queue.  So if their was 100 EF marked packets in the queue they would all be sent before anything else.

---

is there a specific percentage rate-limit also on the Low-Latency Queue so that VoIP-Traffic get`s policed and does facilitate other traffic as well?

[PhilipDAth]

>is there a specific percentage rate-limit also on the Low-Latency Queue so that VoIP-Traffic get`s policed and does facilitate other traffic as well?

No there is not.  The EF queue is always service first until it is empty.

[GIdenJoe]

That's an unknown that has bothered me as well for a long while.

Cisco itself always has recommended using no more than 30% of the total bw for the strict prio queue because you could potentially starve out other queues.

So the 1/7th, 2/7ths, 4/7ths bw guarantee claim is not accurate if you have a llq that can potentially grow to 100% of the bw.

I'd also like to know what happens if you both have the EF46 realtime implicit rule and a high bandwidth priority explicit rule configured for all Voice and Realtime video class.

[whistleblower]

what i am wondering is why is the overall ratio given x/7, is there a reason for this?

Specifying a traffic shaping rule as High, Normal, Low guarantees a certain fraction of the uplink to each priority level. The ratios are as follows:

• High       4/7
• Normal   2/7
• Low       1/7

---

@GIdenJoe wrote:

I'd also like to know what happens if you both have the EF46 realtime implicit rule and a high bandwidth priority explicit rule configured for all Voice and Realtime video class.

---

IMO a really good question which I also asked myself as well and in that context I`d like to know, how can this implicit rule/queue be seen?

In general I`d like to know if/how the respective queues (/w matched- and marked packets, bufferd-packets, dropped-packets, etc.) can be viewed on the dashboard for troubleshooting purpose!

maybe someone here who can answer/clarify this questions?

---

---

[PhilipDAth]

>4. I assume that the “DSCP tagging” setting does nothing other than mark packets with...

DSCP tags are preserved across AutoVPN.  If you tag the packets then when they pop out the other side Meraki switches can treat them different as they get queued to go out specific ports.

I have been told, but not been able to confirm, that the MX15 code copies the DSCP value to the outer AutoVPN header.  Some service providers allow you to buy QoS profiles based on the DSCP values.  But you should consider this as conjecture unless someone is able to confirm that DSCP tags are in fact now copied to the AutoVPN packets.

>9. All of our branch sites use Internet connections (as opposed to MPLS, for example), so once VPN traffic leaves the MX it is encrypted.  I would assume any DSCP tagging on traffic leaving a branch MX would not be visible to any upstream devices until the traffic reaches the hub MX.  Is this correct?

This kinda ties in with my answer above.  Historically the DSCP tags were not visible on the outer AutoVPN tunnel.  This may still be the case.  I had someone tell me that it does now present those tags in MX15 but I have seen no documentation saying this, nor have I had a chance to see a packet capture to confirm it for myself.

[cmr]

---

@PhilipDAth wrote:

I have been told, but not been able to confirm, that the MX15 code copies the DSCP value to the outer AutoVPN header.  Some service providers allow you to buy QoS profiles based on the DSCP values.  But you should consider this as conjecture unless someone is able to confirm that DSCP tags are in fact now copied to the AutoVPN packets.

---

Unfortunately this isn't the case, at least in 15.27.  We had an issue where we were getting high latency over an MPLS circuit that has QoS set on it and it turned out that all AutoVPN traffic was going into the standard class, even EF marked voice.

[PhilipDAth]

>5. The purpose of DSCP tags is to provide a device with instructions on how to treat traffic classes differently in terms of precedence and drop probability.  What do MX appliances actually do as far as processing traffic based on their DSCP tags in terms of precedence and drop probability?

This one is easy - nothing.

Well, with the special exception of EF marked packets for VoIP RTP packets.

>6. If the answer to #5 is that the MX does not process traffic differently based on DSCP tags, does that mean that the MX is limited to simply marking packets with DSCP tags 

Correct.

[GIdenJoe]

@PhilipDAth , the MX should one thing extra than mark DSCP tags and that is putting it in the correct outbound queue (high/med/low) but it would essentially be tail drop in that queue if it is full.

At least the configuration/documentation says so.

And about the policing, you can limit the bandwidth for a traffic type but I'm not sure if for the MX this is also a per flow limit or a total policing limit like you would do on a Cisco ISR.

I also assumed the DSCP tag would be copied to the outer IP header.  Hmm, I'll have to do a test setup with a switch in front of it to verify this.

If it doesn't then I'd be really disappointed.

@whistleblower the reason for the 7'ths is simply because they wanted each queue to be double the previous one.  So low queue = 1, mid = 2 and high = 4.  Add them up and you arrive at 7.

[GIdenJoe]

I have an update on this matter.

Today I had to further check the QoS handling for a customer that does AutoVPN from different offices to a datacenter concentrator mode MX.

These branches have both an internet connection and a MPLS connection that is able to accept pre-marked packets and stick them in four different queues.

At the branch there is an MX84 running 14.40 in NAT mode and in the DC there is also an MX84 running 14.40 in concentrator mode.

After some testing and using different DSCP values than normal for voice I could more easily make out the tagged and non tagged packets by my design in the packet captures.

I made sure the traffic was tagged by the windows pc via GPO and tested it using jabber towards external phone.  That traffic passed through the datacenter.

I first captured on the MS switching using a mirror port.

And then I both captures on the LAN and WAN of the MX both via dashboard and by looping the WAN port of the MX through two external VLAN ports on the switch.  Because I had the feeling the dashboard capture does not always show the same results.

So the NAT mode MX does in fact copy the existing DSCP value of the packet to the outer IP header between both MX'es.  And QoS can be applied.  Even if there is no explicit rule for that traffic.  Default QoS markings is on though.

The MX in the datacenter however does receive the downstream packets from the voice server tagged but it does NOT carry the DSCP value over to the outer IP header.  So this should be categorized as a bug.  The whole idea of doing SD-WAN over MPLS is so you could differentiate between flows even if they are tunneled and put them through the correct queues.

[ww]

@GIdenJoe did you also test what happens  with traffic /dscp tag from spoke-one armed  con- spoke. ?

[GIdenJoe]

I could only test the actual customer I was working with.
Their situation is a one arm conc. in DC and NAT mode spokes all around.

I don't see why you would want a one armed spoke MX in any normal design outside of doing some wireless concentration.

If you'd ever get in a situation like that, please do share 🙂

The experience taught me one thing: Be sure to match your packets one for one before making a conclusion.

And do this by sending equal length packets (like voice) and comparing them to the packets on the WAN interface that also have a size like the internal with 60 or so bytes overhead.

[cmr]

@GIdenJoe Main DC of an SD-WAN solution over MPLS/VPLS, it allows more than 2 WANs at the core. 👍

[PhilipDAth]

>10. We are planning to reconfigure the VPN routing so that the default route at branch sites will be sent directly out to the Internet rather than over the VPN back to the hub ...

If the traffic is going out to the Internet then DSCP marking has no point (unless your ISP is providing some kind of different service based on those tags - that that would not be normal).

[Mikekaleny]

Thank you both @PhilipDAth and @NolanHerring for your responses. I'm surprised the official documentation doesn't mention that the MX is not capable of processing traffic differently based on DSCP tags even though it has the ability to make the appropriate markings. I can't be the only one who initially thought that it was capable of it.