Help with streamlining our network

[DarkRobo]

So I’d like to start off by stating I’m Air Force (Combat Communications)and currently deployed. We inherited a network that is a combination of both Meraki products and older Cisco products (Aironet 1140’s, 2960’s, etc).

Just recently our network has begun to seriously lag and with the holidays around the corner our higher ups want people to be able to call home. I am trying my best to fix this. I have never touched Meraki stuff until this deployment but do hold my CCNA R&S, so I’m not new to Cisco in general.

We have five WAN IP’s that share a 200Mbps throughput to our MX100. Our network serves approximately 500 people who are allowed to bring 2 devices. There are 1400+ unique devices on our network. I’ve already spoken with a Meraki Technical Support Engineer and he recommended new equipment. I agree. My supervision does not. So until I convince my supervision to purchase new equipment, I’m asking for help in utilizing the MX100 to its fullest degree possible.

There is more to the network, but I’d like to see if I can get any help before going forward. I’ll try to reply as much as possible.

Thank you.

[DarrenOC]

Hi @DarkRobo , I’m ex Army (Royal Signals). More than happy to assist you guys whilst you’re out in the field.

So you have a 200Mbps cct connected into a single MX?  The MX100 supports upto 500 clients so you’re way maxed out there.

https://meraki.cisco.com/product-collateral/mx-family-datasheet/?file

Whats downstream?

[DarkRobo]

To give background on that, the entire network was limited to 10Mbps per client up and down, until last week when I got tasked with fixing it. So as of now, there are no limits bandwidth wise on our network. But, we are only pulling 100Mbps internal as of a minute ago. I had seen it go up to 180Mbps at one point then it never did again.

I do hope that’s what you meant by downstream.

[DarrenOC]

Apologies, I’ve re-read your initial post. You’re using legacy 1140 with Catalyst 2960’s.  Your APs went end of life 2 years ago.  Assume you have something like a WLC5508 as the controller?

For starters the MX100 supports up to 500 clients and you’re trying to squeeze over 1400 devices on to it.  There’s a question around your WAN circuit, is it 100Mbps or more?  With all those devices you’re looking at less than 1Mbps per user when all devices connected. That’s quite a contended circuit.

The higher ups don’t want to spend more but you have a maxed out circuit and firewall so it’s going to be laggy.

[DarrenOC]

To make the most of what you have.... you need to restrict the MX100 to 500 clients and if you can up the bandwidth on your internet connection.  

To try and put it into some form of context your 100Mbps circuit (800megabits) divide that by 500 client devices gives you 1.6 megabits per device

[DarkRobo]

I sent you a PM for clarification

[DarrenOC]

Replied.  As per my message, you're very much limited with the MX100 and the number of clients on the network.

[cmr]

@DarrenOC 100Mbits is 100Mbits, not 800...!  If the circuit is 200Mbps and there are 1000 devices then that leads us to 200Mb/1000 devices => 200Kbps per device!

However @DarkRobo all is not lost, what is the bulk of the traffic, if web browsing then perhaps caching might help?

[DarkRobo]

I don’t believe I have caching in. I can implement that tonight and see what happens.

I know @DarrenOC had asked if I had any wireless controllers. Well I just so happen to find two in our supply tent. Would doing anything with 3504 WLC’s help at all? I think it could be easier to convince supervision to upgrade our speed if I could find a way to integrate those with Meraki.

Thoughts?

[cmr]

@DarkRobo the 3504s would only have an affect on the wireless, what are the current controllers or are the 1140s running independently?  Do you perhaps have some newer APs as those ones went end of support over two years ago and don't move beyond release 8.

[DarkRobo]

There are no current controllers in our network. All 1104's run independently off of the switch we supply internet to. The majority of our network consists of supplying a new switch with a trunk port, then setting up a vlan for a wireless access point.

We do have some MR84's in our network, and have 6 more waiting to be installed somewhere. But everything runs to the MX100 through basic switches.

[cmr]

The MR84s are way more capable than 1140s, I'd get them deployed ASAP so at least the client side will be less congested, unless they are for additional areas to be covered?

[cmr]

@DarkRobo how many 1140 APs do you have, if they are near to each other then controlling them with the 3504 will make them work together much better.

[DarkRobo]

We have approximately 30-40 spread out across base.

[cmr]

What licenses do you have on the 3504s?  They can manage up to 150 APs so no worries there.  At least you'll know how many APs there are and be able to give them a decent consistent configuration. 

The latest software that works with 1140s is 8.3.150 I think.

[DarkRobo]

I don't know. They were still in the box and I wasn't able to boot them up just yet. But we have two, and a good chunk of our 1140 APs are on one side of the base.

We can check them out tomorrow.

[DarkRobo]

As far as I know, they are not being used for anything. I have them staged to be installed in the coming weeks. We're waiting for some poles to be constructed so we can place them outside. But they are definitely going to be put up as soon as possible.

[DarrenOC]

Best leave the 3504’s in their boxes as the minimum supported WLC firmware is 8.5.  The 1140’s were only supported upto WLC firmware 8.3.150

https://www.cisco.com/c/en/us/td/docs/wireless/compatibility/matrix/compatibility-matrix.html

[DarkRobo]

Bummer. Back in the box they'll go.

[DarrenOC]

🤣 sorry @DarkRobo.  I’ve been caught out in the past upgrading Cisco WLC’s so that compatibility matrix is your friend

[cmr]

@DarrenOC is an unsupported WLC with unsupported APs better or worse than unsupported individually configured APs?

[DarrenOC]

It just won’t work.  The APs won’t register to the WLC

[jbright]

How is the Catalyst 2960 connected to the MX100?

Does the 2960 have a gigabit ethernet uplink or are they limited to fast ethernet connections? 

If there is only a fast ethernet connection, this is going to be a bottleneck...

[DarkRobo]

So the 2960’s are connected via FastEthernet to the 3650’s, and the 3650’s connect to the MX100.

No 2960 (L2) connects directly to the MX100

[cmr]

@DarkRobo That was a good point by @jbright, I'd connect the 2960s gigabit ports to the 3650s, as at the moment if a 100Mb port is used, then all devices on that switch are limited to a combined throughput of 100Mb.  That could help explain why you hardly ever see over 100Mb of traffic at the firewall as it would have to be balanced out over multiple 2960s to achieve that

Also what uplinks do your 3650s have, how many are there and are they connected together using those ports?

[DarkRobo]

So we're going to be going around to our switches and reconfiguring all of them to have their trunk ports (switch -> switch) moved to Gigabit uplinks. We'll see if that does anything.

I was able to speak with some other supervision on getting an MX250 to work in conjunction with the MX100. In the meantime I'll work on limiting applications and usage across the network to make it a bit better.

Does anyone know how to make muilticast UDP work with the Meraki? I know a huge reason our network bogs down is because the rotation before configured these AverCaster Encoders to stream TCP instead of UDP across our network. I tried my best getting multicast to work on network but ended up bringing it down for about a minute or two when trying to implement it. Had to revert and haven't played around with it since. 

[DarkRobo]

So I was able to get them on board with purchasing an MX250. Now we just need to find a place in the UK/Germany that ships fast. 

[cmr]

@DarrenOC is a UK reseller and sometimes holds MX stock.

[jbright]

So what happened after you changed all of the switch uplinks to the gigabit ports?

Can you max out the internet circuit now?

[DarkRobo]

Well we haven’t been able to get all of them changed, I’d say we’ve gotten about 80% done. But some folks on base are saying the internet is faster for them, so that’s a start.

There were other factors involved in that too. The MR84 antenna placement was wrong in some areas where people would sleep/congregate. They’d be sector antennas instead of omni, or they’d have the antennas pointed in the direction of nothing. So we fixed a lot of that stuff too.

My focus right now is the rest of the uplink ports and finding someone in Germany who sells an MX250.

I’ll keep updating as I go along

[DarkRobo]

So we were able to increase speeds slightly throughout the network with the Gigabit uplinks to the other switches. People have noticed the faster speeds. I never get to use it the way they do, so if they say so I take it with a grain of salt. 

My real test is getting non-buggy phone calls. For some reason when we connect our laptops to the extra ethernet port to our WAN, our WhatsApp/FaceTime/Skype calls are near perfect. But when we connect through the MX100 it becomes jittery and buggy. We drop about every 5th word.

Would anyone know why it's doing that?

[jbright]

It sounds like you need to implement Quality of Service. There is probably a lot of contention for the internet bandwidth.

I would start at the MX100 because it is the easiest of your network devices to configure QOS on.

On the Meraki Dashboard go to Security & SD-WAN > Configure > SD-WAN & Traffic Shaping > Scroll Down to Traffic Shaping Rules on that page and pick enable default traffic shaping rules if it is disable.

At the top of the page under Uplink Configuration, look at WAN1 and try to set the bandwidth settings close to what your actual internet connection download and upload speeds are. Maybe go just slightly higher than what they actually are, but not much. The MX100 is capable of supporting up to 750 megabits of throughput. The firewall needs to understand what kind of internet bandwidth it has to work with so it can make reasonable traffic shaping decisions. After you do this, do some testing on the network voice applications and see if this helps any. If not, then it's time to move on to the switches and eventually the wireless access points. This is not a quick task and it may take some time to tune the network correctly. Don't give up, you should be able to get QOS working well throughout the network and your network users will appreciate it. Let us know how it goes.

[cmr]

@DarkRobo make sure you click on custom and choose accurate upload and download speeds like below:

 

[cmr]

It might also be a good idea to check the Organisation / Summary report view and take a look at the Device Utilisation graph.  There is also a new CPU usage graph, but that doesn't seem to be populating just yet!

 

[DarkRobo]

So I think one of the first things I did was to enable QoS and ensure that all voice traffic was being given priority. That unfortunately hasn't made a difference. Voice traffic is still jittery coming in. We've cleaned up the majority of the lines going from switch to switch, so now I think I'm going to have to start doing the QoS on each switch.

I won't even try to lie about that. I've never had to worry about enabling QoS except for labs in my CCNA R&S. So while I understand what I'm trying to do, I've never actually done it. I'm already reviewing the configuration guide for each switch. So if anyone has any tips or tricks they want to throw my way, I'm all ears.

Thanks.

[cmr]

@DarkRobo I'm not a Cisco IOS expert but I'd start with Auto QoS trust DSCP on all ports.  If the devices at either end are marking the traffic then this should help and it is a simple place to start.

[DarkRobo]

Any reason why enabling/disabling that Auto Qos Trust DSCP would give me a warning stating:

Warning: - removal of service policy will cause inconsistency with GigabitEthernet1/0/6 in ether channel 5
Warning: - removal of service policy will cause inconsistency with GigabitEthernet1/0/10 in ether channel 2
Warning: - removal of service policy will cause inconsistency with GigabitEthernet1/0/11 in ether channel 2
Warning: - removal of service policy will cause inconsistency with GigabitEthernet1/0/12 in ether channel 2
Warning: - removal of service policy will cause inconsistency with GigabitEthernet1/0/14 in ether channel 7
Warning: - removal of service policy will cause inconsistency with GigabitEthernet1/0/15 in ether channel 7
Warning: - removal of service policy will cause inconsistency with GigabitEthernet1/0/21 in ether channel 3
Warning: - removal of service policy will cause inconsistency with GigabitEthernet1/0/22 in ether channel 3
Warning: - removal of service policy will cause inconsistency with GigabitEthernet1/0/23 in ether channel 3

When I enabled it, it gave me WAY more warnings. The network didn't crap out, so I guess that's a good thing.

[cmr]

If you have etherchannels then you need the same settings on the channel as the underlying ports. If you look at the config for port G1/0/6 and the config for ether channel 5 what do both have for QoS settings?

[DarkRobo]

Currently, because I took the command off, they have nothing. I wanted to read up on that warning. Like I mentioned before, it didn't bring down the network, so I'm cool with it.

I did an int-range to ensure I got them all. Some ports just had their protocol down and were taken out of the etherchannel bundle. I'll have to look into that in the morning.

[DarkRobo]

Just so everyone is updated, we were able to get an MX250. It's on the way. I was able to configure all switches to perform auto QoS and some people have noticed a better service, others not so much. So now we wait until the new device gets here.

Any tips on running it in conjunction with the MX100?

[cmr]

@DarkRobo I'd start by simply replacing the MX100 with the MX250, let it bed in and go from there.

One point to note is that the MX250 has SFP+ WAN ports so hopefully you have a transceivers. 

Once bedded in, if needed you can then choose what you might want to break out through the MX100.

[DarkRobo]

What exactly would I need the transceivers for?

[cmr]

I'm guessing your Internet connection uses an RJ45 connector as the MX100 has RJ45 WAN ports.  The MX250 has SFP+ WAN ports so you need to convert that to RJ45, or a fibre connection if your WAN has that option.

[DarkRobo]

Ooooooh!! Yeah, we got about a thousand of those. We're supposed to be doing fiber out here in the future, and we have all the equipment collecting dust. So we'll be fine once it arrives then. I thought for a second it was something I didn't mention that we needed to purchase

[Bruce]

For the WAN1 and WAN2 ports on the MX250, they're SFP+ based. There is no 1000Base-T RJ45 port for these like there is on the MX100. You'll need a MA-SFP-1GB-TX or equivalent (e.g. Cisco GLC-T= or GLC-TE=) for the WAN ports on the MX250 if you want to connect them to an RJ45 cable.

[DarkRobo]

So our network setup looks like this:

MX100 as our gateway

3650 as our “Core Router” (L3) with 

- 4 AFN Encoder for receiving satellite and streamed over the network TCP

- A download server

- Plex servers

Trunked to

Another 3650 (L3) that has direct connection to 7 wireless access points.

that is trunked to 1 3650 (L3) and 2 2960‘s (L2) and a PtP link over a mile away.

that 3650 has 9 wireless access points directly connected to it, plus two cameras etc etc etc etc etc.

It’s literally easier if I sent a picture.