Interference on 2.4 GHz

[jmorphew]

We have a newer Meraki wireless deployment.  The building is a 2-story office area attached to a large warehouse.  Our 2.4 GHz wireless is not working reliably in the warehouse portion of the building, the 5 GHz wireless is working.  I am already working with Meraki support and a local vendor, site surveys have been completed.  So far neither has provided a solution.  Here’s some additional information:

• We are using MR36 and MR44 access points throughout the entire building (warehouse and office)
• Our warehouse access points are mounted at 34 feet, a few feet below ceiling height
• Nearly all access points in our warehouse show 90% or higher utilization on 2.4 GHz
• The minimum power level of our access points in the warehouse was set 29/30 to address some deadspots and low coverage areas.  We have been adjusting this setting, they are currently set to 20/30.
  • We adjusted minimum the power level back down to 6 for testing, but it only reduced our average utilization by 4.5%
• The 2.4 GHz connectivity issue is consistent across all SSIDs

There are problems throughout the warehouse, but to make this easier I will focus on one location.  The attached image is our floorplan in this location.  The red numbers are utilization, the blue numbers are 2.4 GHz channel.  Here's some information about one specific location in our warehouse:

• A portion of our warehouse has racks that are 30 feet high and 8 feet across.  The racks are filled with pallets containing cardboard, wood, metal, and plastic
  • The access points are 19 feet apart East/West (between racking aisles)
  • The access points are 84 feet apart North/South (down the same aisle)
• When a client is using 2.4 GHz and stands where the Green X is located, they have the following issues:
  • Slow internet (1 mbps or less on a 1 Gbps connection)
  • High latency (500-2,000 ms)
  • Dropped connections, no internet
• This client is closest to access point W-21 (green box)
• Access point W-21 is on channel 6
• The average channel Utilization for access point W-21 is 98% (jammed)
• There are hundreds of entries on the ‘Interfering APs’ list for this access point
  • The top 4 interfering BSSIDs belong to neighboring access points WH20, WH19, WH23, WH28 (red boxes)
  • These access points are also on channel 6

I believe our issues are caused by co-channel interference due to a high density of access points in our warehouse.  I also believe the wrong model of access point was chosen for a warehouse deployment, driving higher density and more channel overlap.  I would like help confirming this theory or help identifying any other potential causes.  I would also love suggestions on how to resolve this issue.

 

 

 

[ww]

Try disable some of the ap's 2.4 ghz radio

For example on w26 w28 and w30 and see if things get better in that area

Or Maybe disable all 2.4ghz if you dont need it

[jmorphew]

I planned on disabling 2.4 GHz on the warehouse, then re-enabling it for a few access points.  But I read that Meraki has a hidden network for mesh that would be broadcasted regardless.  So I wasn’t sure if this would be a valid test.  I plan on powering down access points next time I have a downtime window, but it might be a few weeks.  We might be able to get away without 2.4 GHz, but I would like to explore anyway to fix it before giving up.

[ww]

If you dont use mesh then  better turn mesh off. On network wide settings

[PaulMcG]

In any warehouse deployment we always go with external antennas, so in your case it would be a MR46E. 

The location of the APs all in a row aren't helping either.  A proper site survey will indicate where they should be, but I don't see the logic of having them placed as you've indicated.

Power settings aren't helping either.  At a customer with a similar environment to what you describe, we have power settings of 9-14 for 2.4, and 10-17 for 5 and almost all the APs are running on the lower end of the scale.

[jmorphew]

Thanks, this aligns with my understanding as well.

[Ryan_Miles · Accepted solution]

The Warehouse RF profile you have instructs both radio bands to use all data rates and min power of 20dBm. That is likely a big part of the issue here. In a dense warehouse environment you'd want to raise the min bitrate and allow lower TX powers.

As others have mentioned internal antenna APs and AP placement might not be ideal here. Internal antenna APs at 30+ feet is generally not recommended. And I would tend to only see this end of row deployment is directional antennas were being used to shoot the signal down the row (like mounted on a wall pointing down the row).

[jmorphew]

Thanks, I'll ask them to try lowering power and increasing bitrate and let you know how it goes.

[GIdenJoe]

Usually your power levels should be no higher than 14 dBm on 5 GHz and 8 dBm on 2.4 GHz to kind of have the same coverage area.

Your ceiling height is a bit high so you could have made the case for directional antennas.

How many SSID's are you transmitting ( no more than 3 hopefully )?
How high is the minimum bitrate (12Mbps minimum I hope)

Did you check for any non WiFi interference using a spectrum analyser?

[jmorphew]

We are transmitting 5 SSIDs.  Four are required, one was added to troubleshoot connectivity issues with the tablets on our forklifts.  The warehouse profile has a min bit rate of 1 for 2.4 GHz and 6 for 5 GHz.  I will ask the vendor if they used a spectrum analyzer, they were walking around with some device but I think it was just measuring signal strength.

[GIdenJoe]

Ok, it seems like best practices are not followed and you are able to fix alot of the airtime by channging these.
1) if you do not use any 802.11b devices in your network you should be safe to use 12 Mbps as a minimum bitrate, this will alleviate most of your airtime shortage.  If you do still have 802.11b then try 11 Mbps.  (risk: if you don't have enough coverage you might have more dead spots with higher bitrate but your survey should have taken care of this)
2) try getting more done with fewer SSID's.  If you are segmenting onto VLANs using SSID's you're better off using iPSK on one or two SSID's.  iPSK couples each pre shared key with it's own Meraki group policy which can set a custom VLAN and this all on one single SSID 😉

[jmorphew]

Thanks, I will try adjusting the bitrate and temporarily disabling the other SSIDs.  Yes, we had deadspots before.  That's why the vendor adjusted these settings.  

[GIdenJoe]

Ok I hope that works for you without too many deadspots.
If it does improve and you have deadspots you might want to consider doing a validation survey by a professional.  There might be an issue with how the current AP positioning has been done.

Btw: without any user traffic you can have high levels of management frame overhead with low minimum bitrates.  The screenshot below is from wifi professionals showing the effect of 6Mbps minimum rate:  It is even way worse in 1 Mbps but you need the excel sheet for that 😉

[GIdenJoe]

Oh I didn't realise I still have the original excel file.
Here are the calculations for 1Mbps as your situation was:

[jmorphew]

Thanks for this chart, to make sure I have this right.  The 'number of APs on channel' column refers to the number of overlapping APs?  And the overhead percentage is similar to Meraki's 'channel utilization'?  So we have a bit rate of 1, 5 SSIDs, and many overlapping access points.  Which basically makes it impossible to avoid interference.  

[GIdenJoe]

Yes that is correct.  The chart purely talks about AP's that are on the same channel vertically and the amount of SSID's horizontally.  The values are the actual airtime utilization that is eaten up by management frames (these include beacons, probe requests/responses, auth, (re-)assoc requests and responses) without even sending out a single ping.

Normally an airtime of 100% can't even be acheived since it usually tops out around 85%.  All above that are wireless frames very likely to get collisions and by effect retransmissions.

[BlakeRichardson]

A minimum bitrate of 1 means devices will hang on for dear life, if you set a higher bit rate the the AP will cut off devices lower than the minimum forcing them (in theory) to connect to a closer AP. 

As mentioned having the AP's all in a line isn't great, how long is each aisle, you could benefit from every second aisle having an AP in the middle instead of one at each end. 

[jmorphew]

Thank you, this is helpful.  The full length of the aisle is about 200 feet, and there are 3 access points to cover that distance (each end and the middle)

[Agus]

Is there using the same SSID name for all access point? i wondering throughput on 1 AP consume for 1 device (maybe that device downloading something) that make other device connect to that AP not receive enough throughput. 

Thank you

[jmorphew]

All access points broadcast the same 5 SSIDs.  Except in the office where we have the 'warehouse' SSID off.  The data usage on our access points in the warehouse is extremely low, I checked W-21 and it has not gone above 3 Kb/s in the past month.  It's primarily handhelds scanning product, which uses very little bandwidth.  When I look a the clients generating a lot of traffic, non of them are in the warehouse.

[jlarson]

I have read the responses and I am surprised and confused by the fact that nobody has raised the issue of having all the AP's on channel 6.  This is almost guaranteed to cause co-channel interference, and using single channel schemes hasn't as far as I know, been done in ages.  Maybe I'm mistaken, since nobody else has mentioned it.

When I've done warehouse designs, we used directional (120 degree max) antennas at the ends of every other aisle, though those were wider aisles than what you have.    

If you think about how csma/ca works...each time a wifi device is thinking about broadcasting anything, they wait, they listen, and if there aren't any RTS/CTS conversations already in progress, then they will send their own RTS packet.  Since everyone is on the same channel, every access point and client has to wait for every other ap and node to finish their RTS/CTS and then conversation.  If you install directional antennas, this reduces the access points and nodes not in that direction from "seeing" the neighboring conversations.   Combining that with spreading the APs across 1---6---11 reduces the amount of channel utilization.

There was another poster that recommended raising the minimum speed to encourage AP's to drop lower speed connections.  This is absolutely true in the standard office, but my experience is exactly the opposite in warehouses in my experience.   First, the devices typically in use in warehouses are very low bandwidth needs (symbol scanners, etc).  When they get into an area where the signals are "blocked" and SNRs are low, they drop off completely.  Now, in your case, you are actually overloaded on AP's so they probably won't experience a low SNR (other than the co-channel interference) this is probably less of an issue.  

Now, for AP placement...I would suggest, at the least, change to directional antennas on every other aisle.  A 120 degree directional antenna will bleed enough to the sides to cover the neighboring aisle closer to the AP, but most of the signal will be directed down the aisle.   

My suggestion would be to try this in a few aisles and adjust as needed then replicate across the warehouse.  You might have to get creative in the center aisle.  It's already not working, go ahead and experiment with different ideas.

Another consideration is that you can also run into this co-channel problem with 5GHz if you use channel bonding.

Lastly, though the software is stupid expensive, buy airmagnet or something similar and get actual results, not the estimated/extrapolated results you might get from the meraki heatmap.   At the very least, get inSSIDer from Metageek...this will show you the actual realtime 

Good luck.

[GIdenJoe]

Simple answer to your first remark:
Because we assume nobody uses the same channel everywhere we never noticed it on the screenshot 😉

Second if you can't even guarantee 12 Mbps then the survey never was done correctly in the first place.  It's not that hard to achieve 12 Mbps especially with more modern Android based scanners.

I personally prefer Ekahau over Airmagnet due to lack of development for the latter and in Ekahau you can easily offset your coverage results with x amount of dB or just say it uses mobile device coverage.

[jlarson]

GldenJoe,

I haven't used Ekahau...just checked it out and way to rich for my bosses checkbook.  

I respectfully disagree on the 12MBps idea.  On a given channel, on a given AP, in a given area...for warehouses and other areas where handheld scanners are the primary client, low speeds aren't an issue in my opinion.  There literally aren't any high bandwidth devices in use in these spaces.  If they can maintain a solid 1 or 2 MBps link, the users will be completely satisfied.  Even android doesn't send/receive much data when it scans a barcode and refreshes a relatively small screen which isn't graphics intensive.

[GIdenJoe]

Right back at you (respectfully): Aiming for 1 Mbps on a survey is really stupid.  You have to make sure your RF environment can atleast reach 12 Mbps everywhere which isn't that hard btw.

Even with your directional antennas which are a great idea inside aisles on 2.4 GHz you can still have too many AP's dealing with too much unnecessary overhead and have too clingy clients if you use 1 Mbps.  Having 1 Mbps rate means your clients are already at an RSSI of -78 to -80 which is insanely low.  I'm not claiming expertise here but I have done a few surveys and deployments myself in warehouses and have never had any issues with 12 Mbps minimum.  My surveys are done with -67 dBm minimum with an offset of -10 dB.  Of course I'm talking about newer deployments that use android based scanguns with an 802.11ac wifi chip, not some old windows CE that's hanging together with tape 😉

[jmorphew]

Thanks, I'll also check out Ekahau.  All our client hardware is less than 3 years old.  In the warehouse it's primarily handhelds like the Zebra MC3300, running Android.  We have some new tablets on our forklifts that don't stay connected, I will find out the hardware that's being used.

[jlarson]

GldenJoe,

I couldn't agree with you more.  A properly designed system should be planned to deliver -67 at all locations on a given primary AP, and -70 on a second AP if possible.

Then when the install is done, a walk is performed using some kind of laptop tool that shows the realtime signal strength so that there is confidence in the final solution.  

I was just called in to do a troubleshoot on a distribution center with two "floors", though the second floor was open steel grating at about 9'.   In the section where they were having real problems, steel shelving was literally shoulder width apart, floor to ceiling.  They had a small number of new Meraki APs that they had replaced their "old" AP's with.  The employees claimed that the old system was perfect, and the new AP's were regularly dropping connections.

My recommendation was to allow for 802.11b speeds in this area.  802.11ac is great, but sometimes there is more to the story than throughput, and those higher speeds require SNRs that some installations just can't deliver without saturating the space with extremely low power APs.

Just my thoughts.

[jmorphew]

Thank you for the detailed reply and map, that is very helpful!  Our vendor is coming onsite tomorrow to make some of these changes, and the map with directional antennas is perfect for a discussion with their management.  I am going to push the vendor to replace antennas at their expense, but so far they have not admitted the MR36/MR44 were inappropriate for a warehouse or that the placement/density is incorrect.

I don’t think the channels were all set to 6 intentionally, I don’t think the vendor even looked at channels.  Meraki is auto assigning the channel currently.

Thanks for the comment on bonding with 5 GHz, this was disabled a few weeks ago.  We are getting a lot of connectivity complaints from users on 5 GHz devices, but so far I haven’t been able to replicate it on my laptop or find any explanation.  I thought it was an application issue, but now I’m wondering if it’s related to our wireless.  I’ll checkout the AirMagnet, a few thousand dollars is not a problem.  If we have to replace/relocate access points it’s going to cost tens of thousands. 

[PaulMcG]

Meraki have a best practice guide for WIFI designs with recommendations on AP models depending on ceiling height about halfway down the page.  Anything above 8' AP with external antennas are recommended.

https://documentation.meraki.com/Architectures_and_Best_Practices/Cisco_Meraki_Best_Practice_Design/...

[jmorphew]

Thanks, I pointed that article out to our vendor a few months ago and they said it did not apply to us because we don't have a high density of clients.  

[jmorphew]

Quick update.  The vendor is onsite today reviewing everything.  Changing the bitrate from 1 to 11 has reduced the interference from 90%+ (jammed) down to 50%+ (high).  We can now use the 2.4 GHz range, speeds increased from 1-3 mbps to 20-80 mbps in most areas.  Roaming is still not smooth, taking 15-30 seconds in some areas.  

The vendor said they did a site survey using a Ekahu sidekick and it did not show any issues.  I've asked for an output of the results.

[BlakeRichardson]

I don't mean to sound rude but it seems like your vendor needs to find a new trade. The way I read it you have done all of the troubleshooting (the installer should be doing this).

They shouldn't be selling equipment if they aren't familiar with it. 

[jmorphew]

Yes, I wish the vendor was doing more to troubleshoot, and had done better planning upfront.  Working with what I have on this one though.  We disabled mesh and disabled 2.4 GHz on about 20% of the access points and the interference went down further.  Roaming is still not very smooth and a portion of the warehouse is still seeing slower speeds and more connectivity issues.  I think we need to disabled 2.4 GHz on more access points.

We're still getting connection complaints from clients and I noticed in the timeline that they are moving between 2.4 and 5 GHz regularly.  I'm wondering if I just disable 2.4 GHz entirely what the results would be.  We have a high density of access points, wrong model, bad placement, running at high power.  But if the 5 GHz coverage is there, do you think it will be reliable?  Or is this just a losing battle and we need to replace and relocate access point?  Any opinion is appreciated.

[redsector]

We have got in every second line an accesspoint in our warehouses. where not then alternately, on the opposite site. And with patch antennas facing in the shelfe alleys. Running with 12MBit minimum. Autopower. Mixed 2.4GHz and 5GHz. Each line is about 60m (about 200ft) long.