Fireware upgrade from 23.11 > current - General performance, security and stability improvements.

Lagcat
Getting noticed

Fireware upgrade from 23.11 > current - General performance, security and stability improvements.

Hi All

 

i have a site locked down on 23.11 firmware

a few weeks ago we tried upgrading to latest stable (25.13)

3 weeks after the upgrade we had to revert due to outages each week - the issue looks like the strength of the APs now completely overlap and now out roaming is completely off

 

because this is a warehouse environment we have the signal tuned for the devices to influence the roaming over mez floors and racks

 

moving back to 23.11 seems to have resolved the issues but i really want to get to a stable version

 

at the bottom of every firmware there is the general bullet point: General performance, security and stability improvements.

 

i am really interested in what these changes are in the firmware as something is making a fundamental change to the signal strength as i can see it in neighbouring APs as all strength is changing

 

the warehouse and office are using MR42s, MR74s and it seems to change on all of them

 

i am going to visit next week to survey the site after the firmware but from what i can see with the issues and the way the roaming has changed we are talking Meters of difference on the overlap

 

i know 23.11 is really old but the issues were were having with disconnects every few minutes were killing us at the time - and every time we tried to upgrade to 24.x we went back to the same issue

 

we have other warehouse and office on 25.13 not but nothing with setup like this for transmit power

 

 

i know the best after is survey survey survey - which i will be doing next week but upgrading has a huge effect on our configuration

i always made sure the client balancing was disabled

 

sorry for going on and making little sense but i dont like the general fixes statement with no notes

 

Cheers

7 Replies 7
NolanHerring
Kind of a big deal

AutoRF is nice, but never rely on it 100% to do a perfect job. Especially in warehouses. It doesn't sound like you have the RF Profiles tuned in correctly if different firmware version is causing them to flake out. 25.X train though uses the same algorithm for AutoRF, however 26.X train has a new version that is apparently much better from what I've been told (I would still avoid 26.X for now unless you have a reason forcing you to use it).

That being said, it just feels like you need to go back into the RF Profile and make sure that the power min/max settings are where you want them. I personally have them set to exactly the dBm level I want and avoid using a range value. I want the access points running at the power level I designed them to be at. In addition, you might want to verify what channels they are using.

If your using for example, DFS channels, you'll want to verify that your warehouse devices (zebra scanners for example) have their configurations to match exactly what channels your access points are using. Otherwise an AP could be on channel 64 and the scanner isn't scanning that channel so the AP doesn't exist to him.

For warehouses I static my channels and power to be honest. Warehouses have a tendency to turn wireless on its head, so you have to put extra care and attention into the design. Office environments for example, I will let AutoRF do its thing with channels, however I still reign in the power to what I surveyed for, but I don't care about the channels so much in a multi-tenant environment.

Tune the power level. Verify the channels being used by the AP and devices. 25.13 should run solid as it does/has for all my sites.
Nolan Herring | nolanwifi.com
TwitterLinkedIn
Lagcat
Getting noticed

Hello

 

thank you for the reply - at the moment I have static power, static channels and dfs disabled

client roam balancing disabled

 

I have between 10-15% overlap for roaming and -67dbm as my target signal strength 

 

all this seems fine at 23.11

 

devices are 5ghz using unii-1 I believe so I get 4 usable 5ghz channels

and device are set to only scan on these usable channels to keep interference/chat to a minimum

 

and really on 23.11 it is solid

 

problem is the amount of features I am

missing like wireless health, security requirements, firmware fixes - especially with security teams checking versions numbers this sort of thing is highlighted quickly

 

when I upgrade to 25.13 my roaming overlap seems to change to easily 30% and no change has been made to power of channels which is having an affect to the devices

so means I need to survey again multiple times to get this correct.

No firmware updates between this mention a increase of this magnitude but every release has that generic general performance upgrade remark

 

so it just if there is somewhere that shows

In more detail The release note for general performance increases

 

really I want to upgrade and keep on schedule which updates but the fear is everything will move to 26.x and it will all be thrown off again

this warehouse is more than a few hours flight and planning for 24/7 production to play with a required service is not easiest conversation

 

Cheers

 

 

 

 

 

PhilipDAth
Kind of a big deal
Kind of a big deal

>at the moment I have static power, static channels

 

@NolanHerring and I probably have opposite opinions of this, but I am a strong advocate of NOT doing this.  I've never done it with a customer (warehouse, factory or office) and simply don't have problems.  The big problem is that it can't handle changes in the environment.  In this case - it can't even handle a simple firmware change.

 

Ekahau like to peddle the line that using their tool and statically calculating everything once is somehow better than letting the manufacturers own system dynamically calculate it.  In Meraki's case, Ekahau doesn't even have the operating parameters of most of the Meraki APs and antennas available.

By all means use Ekahau to help plan out where to put APs, help select antennas, and perform post-installation surveys to verify coverage - but I wouldn't take it any further.

 

This is my personal opinion.  Many many people will disagree.

 

 

Limiting the AutoRF parameters (as opposed to statically configuring each AP) is a different story.

 

I'd rather put in some extra APs than have the configuration rigidity of using a static configuration.

 

The solution should be able to identify and self resolve problems - without humans having to intervene.

NolanHerring
Kind of a big deal



@Lagcat 

 

I too would prefer to know specifically what the 'general improvements and stability' are in the release notes, so I feel your pain there.

 

You seem to be on the right track with how you have things configured as far as I can tell from here. Are you using directional antennas?

 

Just out of curiosity but how are you determining this 'percentage' of coverage going from 15% to 30% after upgrading the firmware? Are you surveying or using a client device or tool to measure RSSI etc.?  You say you had weeks of issues, what specifically were users reporting was the problem?

 

If somehow the power level you have set (say 11dBm on 25.11), somehow ends up going to 14dBm (which is double FYI), when 25.13 is in use, then I would agree that would/could be an issue. I'm assuming the GUI/dashboard on the AP or on Radio Settings is showing you the actual transmit power still being the same.

 

Is support able to verify that you aren't somehow running into some strange bug where maybe your hardware batch (possibly only yours) is hitting that is causing power to increase unexpectedly?

 

 

Haha @PhilipDAth I don't disagree completely.  =P

 

AutoRF/RRM/Whatever-vendor-flavor, like you said should be able to identify and self resolve problems etc. Honestly they all tend to do a not too bad job at it. Its going to be able to detect changes, and react, far better than I ever could. I use AutoRF myself for most of my deployments, with the exception being warehouses.

 

I will say though that they could always use improvement, no system is perfect and it requires the installer to understand how it works, and adjust/tune the settings so that they operate within the thresholds surveys for. Letting RRM run default out of the box, at least in my experience, has usually never worked out to well with how they end up finally settle (at least on power which is usually my main concern). Channels is meh...as long as they calm down after the first X hours/X days for it to adjust itself.

 

Warehouse though I use static power and static channels. Unless your's differ from mine, the warehouse is usually isolated in a field with nothing too close. So the only wireless in there is mine, and 5GHz only operation for production, I have yet to run into anything (other than itself) that it has to adjust to.  And since it is production, I want it to be as stable as possible (meaning no changes like channels flapping around).

 

Not saying it isn't possible that someone or a vendor could come in and install a wireless camera without me knowing and its crushing say channel 149 for hundreds of feet. However I would classify that under the exception folder.

 

What I have seen though are deployments (specifically with Meraki back in my VAR days) where autoRF was changing channels like its life depended on it, which ended up causing more issues than not. This of course isn't autoRF fault, its just that warehouses in general suck for wireless.  Giant metal box without any actual walls.  You got lots of empty shelves because inventory is fluctuating and all of sudden 20 access points can see each other like they are 15 feet away, and not 60, because the inventory on 10 rows changes from bags of dirt to boxes of feathers or something.

 

 

Ekahau like to peddle the line that using their tool and statically calculating everything once is somehow better than letting the manufacturers own system dynamically calculate it.  In Meraki's case, Ekahau doesn't even have the operating parameters of most of the Meraki APs and antennas available.

By all means use Ekahau to help plan out where to put APs, help select antennas, and perform post-installation surveys to verify coverage - but I wouldn't take it any further.

 

This is my personal opinion.  Many many people will disagree.


 

Not sure about the Ekahau peddling part, but I agree with you if they are saying that. I use it to help me design, save me time onsite, but its not the end-all by any means.

 

Nolan Herring | nolanwifi.com
TwitterLinkedIn
NolanHerring
Kind of a big deal

@PhilipDAth 

Forgot to ask

Have you been using 26.X anywhere yet? I haven't. I heard AutoRF was 'vastly improved' by a source, so I'm curious if you've noticed anything. Nice to hear though they are always working to make it better 😃

Nolan Herring | nolanwifi.com
TwitterLinkedIn
PhilipDAth
Kind of a big deal
Kind of a big deal

Only in our office, which is small, and includes an MR45 (so no choice in the matter).

 

Had zero problems so far.

Lagcat
Getting noticed

Hi Nolan

 

thank you for the reply

 

I am heading to the warehouse next week to survey again and again to validate how the power/performance have changed

 

I use Netscout survey Pro and we have Zebra devices which has some SSIDer type app giving RSSI details

I had spent weeks on the deployment previously getting the roaming to a reasonable level I I worked out my 10-15%  with multiple surveys for single APs showing the SSID and in most cases tape/chalk on the floors mapping out dbm and roaming variables for the devices

 

the warehouse has moving stock and liquids/foil in most places including a 3 level warehouse floor so I had to get my bleed levels as accurate as possible

 

if I receive a call regarding wireless they can tell me the floor/isle and I know what AP they should be on/joining/leaving as I know the boundaries I have in place to keep signal and roaming conditions

 

what I was seeing after the upgrade were devices far passed the boundaries configured for roaming still attached to the same AP also at the time of roam they were now hopping between multiple APs within 30-50secs of joining

I had our remote technical guy take a walk to different areas with a Zebra device and the signal was -3 to -5  better than before which in most cases would be a good things but the size of this warehouse and floors separated by metal grate - this adds lots of interference

with the power increase this is bringing the possibility of overlapping channels closer today in places of high density so this is potentially now the issue

 

 

the constant roaming was effecting the performance of the devices to make them unusable - after the downgrade it has been perfect again

 

at first I though I had forgot to disable client balancing but it was already disabled

 

I think I am going to be spending a long time troubleshooting this again - I don't mind it as its fun work mapping It all out I just was not expecting the impact

 

when it comes to auto power and auto channel this went badly for us practicing back on install in this warehouse - it was like a wave of change every 20-30mins with the amount of reflection/scattering and the thin floors the APs really were not handling it well

 

we have other warehouses/offices running on auto no problems so I am have nothing against letting the software do its job - but everywhere always ends up with a problem site

 

Cheers

Get notified when there are additional replies to this discussion.
Welcome to the Meraki Community!
To start contributing, simply sign in with your Cisco account. If you don't yet have a Cisco account, you can sign up.
Labels