9166I and 9300 Switches

[JohnEustace]

Trying to get some 9166I access points online today.  Ceiling mounted through Cat6E patch cords (x 2 ) and structured cabling.  They power on but don't do anything else (amber LED which never reaches "rainbowing").  If I remove them and patch them straight into the switches they get DHCP, register with the dashboard and pick up their config without issue.  Cabling fault you might think.  However, if I then put them back on the ceiling they register with the dashboard again just fine and the switch ports come up at 5Gbps.  It's almost as if the PoE at the ceiling location isn't quite enough to get them to fully boot but once they're registered they're fine.  In situ they are drawing way less than 60W anyway so I'm confused.

Anyone else seen this?

Workaround is to "stage" them direct into the switch and then move to the final location but that's a lot more effort.

[cmr]

How long is the structured cabling and is that Cat6e as well?  Do you have a network / PoE tester that can check maximum draw and throughput from the remote end?  We have generally installed our CW9166s by staging themas we like to leave the boxes back of house, but I'm sure we have installed some direct from the box and on 802.11at switches...

[JohnEustace]

Don't have a PoE tester (other than confirming its there) and I don't know the length of the cable runs unfortunately but I don't think they're excessive. The strangest thing for me is that they do work fine in those same locations if they've been staged with a short patch lead already.

[PhilipDAth]

When you went to bring them online - was the first time while plugged into the ceiling?  If so, they'll almost certainly have to do a firmware change, so that first boot will take a long time.  Did you give them 20 mins?

[JohnEustace]

Yes they had been plugged in all day. Nothing more than an amber LED. Took a couple down an plugged them straight in to the access layer switch and they upgraded and came on line straight away.

[JohnEustace]

Then stayed that way when I moved them back.

[Paccers]

What is returned from an interface TDR test on the 9300 (if not running in Meraki-managed mode) when the APs are patched in to the ceiling ports? If they are fine when patched in to the switch direct it does suggest an issue with the structured cable runs

[JohnEustace]

The cable test returns "down" when it's clearly not so I'm not sure that's functioning very well in the 9300 range. Unless the test doesn't like the Cat6a cabling somehow.

[cmr]

The test does work, definitely a cabling issue!

[JohnEustace]

It's starting to look like the installers might not have pushed the RJ45s fully home on the APs.  The Cat6a patch cords are a bit of a monster and the connector needs a bit of a wiggle before it clicks into place.  The ones we took down to stage were the ones we refitted ourselves so connected them properly.  Lesson for the day is if you want something doing properly you have to do it yourself.

[JohnEustace]

Tried a Cat5e patch for the final drop at the AP end and that worked too.  Then it also came up ok when we reconnected the Cat6a patch.  A lot are now connecting just by reseating the AP Cat6a drop cable.  My latest theory is it's not a good idea to leave the outlets patched in a dusty building site environment for weeks before you try and plug the APs in.

[cmr]

Dirty contacts will definitely play havoc with PoE.  Some contact cleaner sounds in order!

[JohnEustace]

Ready for the next chapter?  Ok here goes.  We got all but 3 APs (of 46) up and running last night.  Turned off all the switches overnight so they don't blow plaster dust through them unnecessarily.  Turned on the switches this morning and only a handful of the APs reappeared with most showing PoE drawn on the switch port but no link light.  Every AP has two Cat6a outlets.  One for the 5Gbps ethernet and one for the console cable and both are connected.  Now for the weird bit.  If we remove the console cable connector and reboot the switch the AP comes up just fine.  Can't believe that's the solution but it seems to work.  I hope this isn't documented somewhere.  I don't really understand how just having a cable plugged in to the console port can affect the boot but it is clearly causing problems.  There is no laptop on the end of the console port it's just patched through to the wiriing closet frame.

[Paccers]

Interesting, if you're running the 9166's in Meraki persona the console port isn't usable anyway, only active when configured in DNA persona

[GIdenJoe]

Well you can't write any commands but you CAN see the boot up messages.  Sometimes there is a clue in there about what could be going on.

The CLI promt then simply shows <Meraki> and you can't issue any commands.

[JohnEustace]

We've disconnected all the console ports on the APs and turned off all the switches again overnight.  In the morning we get to find out if we've discovered a new Meraki feature.  i.e. the ability to prevet an AP booting just by inserting something into the console port.

[JohnEustace · Accepted solution]

All APs up without issue this morning after removing all the console port plugs yesterday and powering down the switches overnight.

So the accepted solution is don't boot the APs with something plugged in to the console port.

[CharlieCrackle]

I can confirm I have seen the same behavior.   3 x 9300 stack with aps in different stack members   APS would not come on line (the switch would show that the POE supplied and the POE requested was lower)   AP have orange light. unplug the long console via the building cabling and all AP would come on line.  (you could plug the console back in and it would continue to work)   problem with leaving the console plugged in after boot was if a software upgrade occurred the AP would not come back after reloading unless console cable removed and plugged back in.   Opened case Meraki unable to reproduce.     ONLY solution was to unplug the console cables on all APS.

[CharlieCrackle]

This may be related  as it is the same hardware as Cisco..  (having same issue with Cisco hardware in CAPWAP mode)

https://bst.cisco.com/bugsearch/bug/CSCwb86411?rfs=qvlogin

Symptom:Wifi 6/6E AP boot hangs when the console cable is disconnected from the console server end, while the AP side of the console port is still connected and cable length is ~200ft

Workaround:Terminate the far end of the console cable in a DTE (PC COM port, terminal server port, etc.)

Made a little test cable....
Given console port pinouts: RJ-45 DTE Pinouts RTS(1), DTR(2), TXD (3), GND(4), GND(5), RXD (6), DSR(7), CTS(8) at the jack side, strapping 1 to 8, 2 to 7, 3 to 6, 4 to 5.