Ethernet uplink speed degraded

[Dave_B]

Hello,

I have an Ap (MR46E) that has an alert of "Ethernet uplink speed degraded".

I've checked the cable by investigating the port...no errors, CRCs, drops...cable seems good.

I've bounced the port.

I have the port set for auto/auto and the port is showing 100/full.

It's firmware is fully updated.

I've set the port for 1000/full, but once I set the speed to 1000, the AP drops and the port shows not connected.

Port is showing power is still being issued with 30w.

Not sure what is happening...

This Ap has been running fine for over a year and this just started happening a week ago.

Any ideas???

[cmr]

Does the cable test show all pairs are good?  If you connect another device using the same cabling and switch port, does that come up at 1Gbps?

[Dave_B]

well I didn't do a cable test per se`...I just checked the port using cli command sh int gi0/x.

port shows clean.

I actually just noticed a few others showing the same alerts from several different sites. I have 5 locations around the globe.

[RWelch]

I had this same scenario occur yesterday with one MR36 with nothing changing in the equation.  I believe it's possibly attributed to one AP that is rarely or seldomly used.  Cables are good, firmware all up-to-date.

 

[RWelch]

I just fixed my MR36 that had a similar issue as you described above.  I turned off the port and turned it back on.  Nothing else in the equation changed.  Not sure what caused it to change to 1OOMB full duplex from 1Gig auto negotiate.  But it fixed mine.

[Dave_B]

just tried bouncing the port on 2 different APs but no joy.

[RWelch]

How about disabling the port for 60-90 seconds then enabling it?  That's what worked for me.  Hope it works for you too.

[TEAM-ind]

This worked for me, as well.

[BlakeRichardson]

Are you using Cat6, Cat5e or Cat5?  

If it's connected via structured cabling how old is that cabling?  I would suggest getting a cable installer that has a decent tester like a Fluke branded tester to have a look, it could be a crushed wire, bad resistance etc that could be causing the problem. It's not as simple as getting a link and speed negotiation.

[RichardE]

I have had this error on 5 x MR46 access points, I only upgraded to firmware MR30.7 a few weeks ago. This is the 1st time I have seen this error and the APs have been in place for years. I tried flapping the switch port on 1 AP and left it off for 2 minutes, but still have the same problem. POE on the switch port is drawing 30W class 4. 

I do not think it is a cabling issue to see it on so many APs in a short period.

[cmr]

@RichardE if they are on different networks, you could try reverting one to the previous version to see if 30.7 is the cause.  If the dashboard doesn't let you go back, support can do it for you.

[ancile]

I have the similar issue as you that happen in my environment in this couple of week, my AP is C9162 and running on 2.5Gb link speed. My c9300 multigiga switch register to meraki portal for monitoring purpose and I receive alert from meraki the link speed changed to 1Gb. I have use the cable tester Fluke to inspect from source to end and the result was pass. Need to manually shut and no shut the interface to recover.

I haven't try to fixed the speed negotiation in the switchport as to me this is not a practical solutions.

I have open TAC case with meraki TAC and the first thing he get me to do try is take down the AP and connect with a new patch cable from the switch to the AP instead of through the copper panel and monitor. 😞

[cmr]

@ancile what is the cabling standard used from switch to AP?  i.e. 1m shielded CAT6 patch lead - 27m unshielded CAT5e structured cabling - 5m shielded CAT6a patch lead

[ancile]

switch-1m unshielded CAT6A patch cable -> patch panel -> shielded CAT6A backbone -> AP

[cmr]

Hi @ancile is the solid core structured cabling terminated in an rj-45 plug at the AP end?

[ancile]

Yea, at the end of the cable terminate with shielded RJ45 modular and connect to AP

[cmr]

Sounds like at least the cable standards are all good.  You could try a shielded CAT6a patch lead, just to be safe.  Not that it should overly matter, but how long is the structured cabling run?

[ancile]

The farthest AP is less then 80meter, some AP just 20-30 meter.

[cmr]

20-30m definitely should be fine!  Have you got a good cable tester that you could put on a link?  How many APs do you have with this problem?

[RichardE]

I have the same issue with some MR46 and MR76 access points since upgrading to MR30.7 firmware weeks ago. The access points have been in place for a long time with no issues. I am going to try upgrading to MR30.7.1 just released though I did have success in shutting down the switch port for a MR76 with the issue for 3 minutes and when I open the port and the access point initialised it was back up to 1Gb full duplex connection again.

[WGPete]

I tried that with one of my APs, come back up with no error then it reappeared after about 30 minutes so keen to know if the errors re-appear for you. Such a strange issue! Waiting for Meraki to come back to me with some advice.

[WGPete]

I also started getting those errors on 16 access points acros a 52 acre site. The issue is affecting APs on different swiches in different locations, on two completely seperate physical networks. All got the error at exactly the same time (13th November at 11:01).

Strangely, some access points on the same switches do not have this error. I am unable to find a pattern. Restarted switches, access points, cycle POE ports but nothing seems to resolve it. This is the only post I have found recently and seems strange to be affecting different organisations.

[cmr]

@WGPete , did you have any firmware changes that coincide with that happening?

[WGPete]

No, I did firmware updates after the issue occured to see if that helped so rolling back those this morning and no others show over the last 30 days.

[cmr]

That is definitely odd!

[WGPete]

Yep, really is! I am actually replacing one of the switches tonight with a new Aruba switch so will be interesting to see if that fixes the issue with that affected AP.

[cmr]

Good test, what switch was in there?

[WGPete]

Currently we hjave a HP 2530 48 port switch being replaced with an Aruba 2930F. The current one isn't POE so we are using injectors for that AP.

[WGPete]

So, the access point that connects to the switch I swapped out no longer has the error on it. Whether that was due to it being off for the duration of the swap out (about 30 minutes) like some other posts have menitoned or the new switch it is hard to say but that's one less AP with the problem. I am going to try and find another affected AP that I can take down for about 30 miniutes today to see if it clears the issue.

[BananaCO]

I've encountered a similar issue although its complicated slightly by some changes within my organisation. 

We had 20 Meraki AP's running perfectly happy across a large site. We've recently closed the site and kept two of the MR46's to use in our new much smaller office suite. 

I've now setup the two MR46's with cabling direct to a Cisco Catalyst 9200 switch and they were running without issue for a couple of weeks and then on November 19th (i think) they started to complain about the link speed. I've checked the cabling, reset the AP's, forced the speed on the port. Nothing seemed to fix it.  I haven't been able to bounce the switch but I'll need to wait for an opportunity to do that.

[BillMcC]

I just now noticed this on 5 MR42's. They are running 30.71 as well. I did not notice this error until I had to fix the low power error they were alerting beforehand. 

On my Cisco switches I had to do the following. 

cdp holdtime 30

And for each interface that was giving the low power alert i made the following change. 

lldp transmit
lldp receive
power inline static max 30000
shut

10 second wait

no shut

______

Now I am seeing the 100mb degraded error on 4 of them. 

Is the new firmware buggered? 

[BillMcC]

Well. After rolling back to 30.7 and my 30 AP's rebooting I have 4 more that are in low power mode. I have a feeling if I apply the static power fixes on my switches they will fall back to 100mbps as the others did. Going to get a ticket in with Meraki today as the AP's are on 3 different switches but showing the same behavior.  

[IPPLTechSupport]

Upgraded to the latest firmware - 31.1.5.1 - from 30.7 last night and one of my CW9162I access points is doing this. Prior to the upgrade, everything was fine. Rebooted the access point and cycled the port on the switch (Aruba 2930F) but it is still negotiating at the degraded speed.

I have seven other of these access points, plus two MR76, and all those are running fine after the upgrade.

[Joemama1000]

Hello, 

Any good news? We are having same problem with MR76. 

[RichardE]

I am having the same issue on MR46 models as well and it was happening on firmware 30.7 so we tried updating to MR 31.1.5.1 but no change. Also tried rebooting APs and powering of Cisco POE switch ports for 5 minutes no change. The only thing we have not tried yet is a factory reset of an impacted AP as they are on remote sites. If anyone can try that and let me know if it has any impact.

[Joemama1000]

Hello Richard, 

Did you have chance to check the ethernet cable?

Cisco said When the 'Ethernet Uplink' alert displays, this usually indicates a physical fault (often a cable) causing the port on the MR to negotiate at a reduced rate.

We replaced the cable, which resloved the issue for one MR6 . However, the problem persists with another AP. 

Factory reset didnt work as well. 

[Messy]

I look after several hundred APs - I see the 100mbps issue all the time. It's almost always a cable issue and fixed by replacing the cable. 

The "low power mode" is a bug - i suddenly started seeing it on dozens of APs - I asked support what was going on and they said it was a known problem, something to do with the CDP messages that they use to communicate their status.

Its annoying but not really a problem, since its not really in low power mode, and even if it was - that mode doesn't actually do anything. I think it turns of bluetooth or something, but it still operates at full tx power etc - wifi not rly effected. 

[cmr]

Low power mode can disable the scanning radio as well, that makes the system less intelligent...

[Macguy]

I've seen this "low power mode" for a long time now.  We have Aruba switches and I've tried various things on the switch end to try and resolve.  The only fix I've had success with is physically unplugging cable from switch and plugging back in.  Sounds dumb but it works every time (same port).  Reboot AP doesn't fix, nor does disable/enable PoE.  Really weird, on the switch end it has the correct power class and power utilization.  For whatever reason, the dashboard claims low power mode.  Glad its acknowledged by Meraki as a bug. 

[Macguy]

So I was mistaken... disable/enable PoE is a fix.  We just had 7 AP's in low power mode from a power outage over the weekend.  Also seeing random AP's with Ethernet Uplink Speed Degraded on a daily basis.  A reboot fixes that issue.  Annoying for sure. 

[Balcarras]

1/3 of our previously totally stable AP's (mostly MR33's) are now showing as "Ethernet uplink speed degraded" I can't believe simultaneously 20 AP's have developed a cabling fault!

They were all upgraded to MR 30.7.1 firmware on January 27, 2025 between 4:00 AM and 6:00 AM GMT

Anyone got a fix that does not revolve around physically disconnecting and rebooting AP's??

[Paccers]

Roll back to 30.7.1?

[ExeterUnified]

Just checking in to see if there has been a resolution to this problem. We replaced all of our Ruckus AP with Meraki AP MR57's.  In the 10 years we had the Ruckus, not once did I see a speed degradation. At first our Meraki's were fine. Then firmware update 30.whatever came out and the speed degraded issue started happening.

Did everything. Checked and changed cabling. We are using MS350-24X switches with most of our APs and MS225-48FP switches with the rest. 

Unplugging and replugging isn't going to be an acceptable fix. I truly believe that it has something to do with the 30.xx.xx and up firmware updates. 

What's strange is that it will show the alert, then magically they will show running at full speed for a while, then go to 100mbs for a bit. It's just annoying more than anything.

I don't want to reinstall 200+ Ruckus APs again, but if this doesn't get resolved soon that may be an option until the issue is fixed.

[hechoinlmusd1]

I have to agree with your assessment.  We were running on version MR 29.XX and everything was fine.  The Meraki APs all negotiated to either 1000 or 5000 Gbps depending on the Catalyst switch model.  We held off upgrading to version MR 30.XXX because of 802.1x problems that Meraki Support confirmed.  We upgraded straight to latest version MR 31.1.5.1 last month.  Everything was running fine up until this past week.  We then had multiple MR56s drop their Ethernet links to 100 Mbps with Degraded Status.  Looking at the forums, we have power-cycled the switchports and also power-cycled the Catalyst switches.  We have even shut down the APs for half-hour as suggested and powered them back up.   50% of the time, the APs would come back to 1000 or 5000.  The rest of the APs stayed with 100 Degraded.  We checked the cabling and there are not errors.  Something from the latest firmware is definitely causing all these Ethernet speeds to be degraded.

[Balcarras]

If we set the switch speed manually on our 2530-48G Switches to 1000Full then the AP's run again at 1000 speed, seemingly without issue.

Spoke too soon - weirdly some AP's come back up fine and others become unreachable.

Same type of AP plugged in to the same switch!

[ExeterUnified]

Yup. Thats what we see too.

[Balcarras]

This shows how random the issue is....

[Davey_Hayes]

Hi All

Just to add to this post , now seeing this on a fresh install that was installed in Dec 24, Mixture or MR46 , MR46E & MR86 running Firmware 30.7.1.

This issue has just started within the last week , this was perfect at the start of the year, which is  suspicious , Random AP's on different switches , different switch ASCIC's 

Low Power mode also has also been a problem @ time of install and the fix was physically disconnect AP for 10 mins , reconnect and Low Power alert disappears.

[Balcarras]

We're going to look at rolling back to a previous firmware 😞

[WGPete]

Just an update with our issue, one thing we did notice was that some of our APs were patched to swtiches that only had 100MB speeds (charity so limited with our hardware upgrades :)). All our APs plugged into 1000MB port switches do not have this issue. Something worth checking as well.

[Balcarras]

We have AP's plugged in to the same switch, some now running at 100! and some still running at 1000.

[Sladie]

I have started seeing this issue, not sure when it started. 5 out of 91 APs different switches/offices. 2 are MR33s and 2 are MR36s. Firmware is 30.71. We had to put in a spare MR32 as a temporary measure and this is also showing same Ethernet uplink speed degraded issue. Tried all solutions including recycling port, recycling switch, forcing auto/1000 etc. Can sometimes fix the problem temporarily but returns. This is something that has changed as I've not see this issue over the years we have been running Meraki APs. APs on the same switch don't show the issue.

[Balcarras]

It surely has to be a firmware issue, has anyone been able to speak to Meraki to confirm the best thing to do is either roll back or await a firmware fix?

[Sladie]

Weird how we are seeing it on an old MR32 reintroduced back into the network as a temporary measure running MR26.8.3. Update available for this AP as it's been off the network for a while so I'll update it and see what happens. MR32 are EOL so not a good test but will be interested to see if it changes anything.

[Davey_Hayes]

Have logged a call/case with support , explained that this issue has happened during w/c 10/2/2025 , was perfect prior to this , which isn't ideal , I know the response will be upgrade , which really isn't good enough as the upgrades are constant pain and you just don't know what else will stop/start working...

The site in question has a bespoke mesh setup for Robotic machinery which is a crucial function , so hoping this issue doesn't spread to these AP's 

[Nick_TM]

Thought this might help - Meraki Support said today on the matter;

"We have a known issue that is very similar, but raised specifically for MR46. Please allow me some time to reach put to out product specialist team explaining we are seeing the same issue in multiple models."

[Sladie]

Thanks for that. It is definitely an issue across multiple models and firmware versions so good to know Meraki support are looking into it.

[Sladie]

Update on our support ticket with Meraki is to check the cables 🙄

[Sladie]

I have also reported to Meraki support. I would recommend we all do the same thing to bring some focus to this issue.

[gmartine]

@Nick_TMdid you get an answer?

[Nick_TM]

Not as yet. They are continually trying to get me to test and replace cables which yield no change which is getting unacceptable. I have escalated this to our Cisco TAM and they are engaging the larger team.

They cant explain and keep ignoring that fact that multiple AP's across multiple sites all developed a 'cable problem' at the same second on the same day after admitting they have a known issue on the MR46.

[Garry_G]

Has anyone tried if the latest firmware fixed it?

MR 31.1.6
Released Feb 18, 2025

[Sladie]

I'd also be interested to know as we are at 30.7.1 and waiting to move to 31.1.5.1 which I'm assuming doesn't fix the issue?

[hechoinlmusd1]

I noticed version MR 31.1.6 was available as a Release Candidate.  I upgraded a few wireless access points to version MR 31.1.6 and it made things worse.  The Meraki access points would not pick up a DHCP address and could not go into Gateway Mode. 

I had to revert back to MR 31.1.5.1 and the Meraki access points were able to pick up DHCP addresses again, going into Gateway Mode normally.

Ethernet is still degraded.  Meraki Support has acknowledged the bug and escalated to their Dev team.  In the meantime, Support is saying to go back to version MR 30.6

Will try that soon.

[RWelch]

Curious what AP model you upgraded?  I've upgraded MR36s (1Gig), MR46s (2.5Gig) and MR56s (2.5Gig) to MR31.1.6 without any issues/problems.

[hechoinlmusd1]

We have MR46s, MR53s, MR56s, MR74s, MR76s, and MR86s, with a huge majority of them being MR56s.  Almost all of the APs showing degraded Ethernet are the MR56s.

Just talked to Meraki Support.  I have two WLAN networks scheduled to be downgraded to version MR 30.6 tonight per Dev Team's and Support's recommendation.  We'll see if this helps.

[RWelch]

@hechoinlmusd1 Did rolling back to MR30.6 help bring back things to "normal?"  Or too early yet to tell?

[hechoinlmusd1]

Rolling back to MR 30.6 did not help.  Talked to Support and will be downgrading same two WLAN networks to version MR 29.7.1

[Balcarras]

Roll back to 30.6 also did not fix the issue for us.

One third of all of our AP's are running at 100 Mbps 😞

[Benrue]

We are also experiencing the same issue and i'm thinking of maybe downgrading to version MR 29 - do you have any luck on this?

[gmartine]

@hechoinlmusd1 was this issue fixed for you with your rollback to 29.71?

[hechoinlmusd]

On some of our WLAN networks, rolled back to 29.7.1

After rolling back, 50% of the APs that were at 100 Mb previously are now at full-Gig, so rolling back helped there.

The other APs that are stuck at 100Mb, we'll have to re-check the cabling, keystone jacks, patch-cables, etc.

[SDM]

We currently have several MR36 running 30.7.1 that on the 18th have now got the degraded message, they are connected to a mixture of 9000 and 2960 switches. I have pinned the port to 100mb full duplex and rebooted but it has not made any difference. This is the second time we have had this issue, changing the speed from auto changed it last time, but has not worked this time

[Erik2]

We are seeing this issue as well - MR46 31.1.5

[Balcarras]

Our 3rd party support company have gathered some Wireshark traces which are being sent to Meraki.

We have two AP's off one switch out of six which are reporting the " ethernet_negotiation_failure" - at the end of his capture we unplugged the two problematic AP's, one came back at 1000Mbps for around twenty minutes but has now dropped back to 100 😞

If this was just a cabling issue, surely after a reboot it would negotiate down to 100 straight away?

Is anyone getting anywhere with Meraki on this issue?

[Sladie]

Nothing of any substance from Meraki support so far.

[Erik2]

Here is my reply from Meraki.  No admittance...

Thanks for reaching out on this one.

I can see that while 8 devices are negotiated at 100Mbps, this looked to have occured some time ago:

As the link negotiation occurs on the initial connection the devices have remained negotiated at 100Mbps. As to why, we really cannot say due to the sime since the last events listed above as well as requiring visibility on the negotiation at the time from the switch.

I would recommend first reconnecting/bouncing the switch ports, should the issue remain perform cabling testing to ensure that the full length of cabling including any patch paneling is operating correctly.


Kind regards,

[Balcarras]

When we disconnect devices sometimes they come back at 1000Mbps but within 20 minutes drop back down to 100, this is happening as you can see in this thread to multiple customers so it's HIGHLY unlikely this is a coincidence and a failing of multiple cables simultaneously.

[Sladie]

I agree. We had a complete site power down for electrical maintenance at the weekend where one of the APs in error is connected. When power comes back the AP is still in error. This would have powered off/on the HP switch and AP. Power was off for over 1 hour. Has anyone had a verified end to end cable test on one of the erroring APs which passed so we can dispute the responses from Meraki support which seem to be deflecting what the real issue is?

[Erik2]

I am working on getting my hands on a cable tester that can verify this.

[Balcarras]

Could the issue be that Meraki "fixed" a bug whereby it was not reporting the " ethernet_negotiation_failure " and now that it is, we're all seeing issues with AP's that were previously hidden.

Does anyone have any evidence before the alert become prevalent that the AP's were in fact connecting at 1000Mbps?

[AndyF]

I think our case "might" be following your theory Balcarras.  We have a rather large deployment of Meraki access points of over 4000 access point in around 400 buildings.  These include various models of Meraki access points and various firmware versions.  Most are on either Meraki or Cisco Catalyst switches.  We recently started having this message appear at a number of our sites on some of the access points.  I have had our team validate two buildings so far where this message suddenly appeared on some of our WAPs.  In these particular cases they were MR46 and CW 9166 models.  In both of these buildings it turns out the cabling did indeed have issues that were serving the access points that were reporting.  Cable tests from switches showed issues and were then confirmed onsite with a NetAlly device. Our techs re-punched cables and the WAPs came back and checked in at top speeds.  In our case so far it seems to be bringing light to an issue we had not recognized that we had on some of our cabling.  I'm also wondering if this is something that Meraki recently changed/fixed in its alerting.  I will update if I see any deviation from this.

[gmartine]

@AndyF in my case I had only one MR36 reporting this issue.  It is a Meraki switch in my case.  The cable test built into dashboard passed no problem.  I had a tech sent to the site, he replaced replaced a cat6 termination and now the link negotiates to 1Gbps.  I like your theory 🙂

[CRCError]

We are having similar issues with MR46 - I have tried moving them over to known working lines; switching ports; factory reset; firmware rollback; changing patch cables; shut/no shut.

I have also tested lines running to APs; cable tester shows pass.

[mak2018]

See this on our MR36s running 30.7.1 as well and no changes on our side and no way all the cables are bad for all the APs reporting the issue. Tried shut/no-shut and hardcoding to 1000 but neither worked.  APs com back up at 100Mbps.    No way all of us, all of the sudden, have bad cabling, it is surely an issue with the AP firmware.  

Doesn't seem to be an issue for our MR56s.

GigabitEthernet1/0/26 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is b44c.90c4.7a9a (bia b44c.90c4.7a9a)
  Description: Meraki AP-M36
  MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX

[Sladie]

Have swapped out patch cables on 2 of the reporting APs which has "fixed" the problem but monitoring as expect it to return. Now have 1 new AP reporting the same issue. Currently have 3 APs reporting the issue, 1 x MR32, 1 x MR33 and 1 x MR36 on 26.8.3 and 30.70.1.

[Ramu]

We are also having a similar issue with our two MR36s that started few days ago, both using firmware version 31.1.5.1

It seems something wrong on Meraki software.

[sam-comstock]

The issue clearly began after the recent firmware updates. We have multiple sites with various switches, and we've noticed the degraded speed error popping up on these MR WAPs. I physically tested the lines using a cable tester, and the Cat6 cables pass the test every time. Additionally, I replaced the affected WAPs with Ubiquiti WAPs, and they’re achieving the full gigabit speed. We’ve reported the issue to Meraki, but unfortunately, there has been no resolution for over two weeks.

[gmartine]

@sam-comstockI have a case open with them and here is the latest I just got.

"Thank you for providing this information. At this time, we have no confirmed reports of link negotiation issues forwarded to our engineering team that match what has been observed in your network so far. However, in an effort to further troubleshoot this issue could you please give our 24/7 support hotline a call when someone is available onsite? We would like to take packet captures and check back end logs simultaneously while the AP is connected to the switch port to monitor the link negotiation process. From there, we will be able to fully determine if this matches a reported negotiation issue."

I called them back but they wanted me to physically disconnect the cable while they take some packet capture.  I am not close to my site then I couldn't do it.  I am not sure if this is something support has asked anyone else.  In the other hand, there is a reason why this thread has been seeing more than 16,000 times 🙂

[Balcarras]

We've sent them packet captures.

On the 26th of Feb! and have heard nothing 😞

It's getting ridiculous now

[gmartine]

@Balcarrasif your setup is large-scale, make sure you involve your Cisco account manager.  There seems to be a lot of large-scale setups impacted by this bug. 

[KFoster]

I am seeing similar behavior (auto neg at 100/full) for new CW9164I attached to a C9200 switch. 

Created case #12847892 with Meraki support to look into this further.

[Richardhlc]

Not sure if related,

one of MR46E had same issue, we noticed the cable is using T586A on both end, resolved the issue by changing to T586B.

[Sladie]

We have swapped 2 MR36s around, one was alerting the other was not. So far after 24 hours both are still ok and are not alerting. Will continue to monitor.

[Balcarras]

We've taken a problematic AP out of the room it was in and plugged it directly in to the same switch & port

(ProCurve Switch 2520G-24-PoE / Port 3)

We used a brand new Cat6 cable.

This is bypassing the existing patch cables, the structured cabling

The results:

"100 Mbit, full duplex"

Sigh

[cmr]

@Balcarras what firmware are you using.  I had my first experience of this yesterday after a power outage, where I noticed one AP was stuck at 100Mb.  I upgraded from 31.1.5.1 to 31.1.6 and it fixed it...

[Balcarras]

MR 30.6

We downgraded from the latest version in an effort to fix this problem affecting 1/3 of all our AP's.

[Sladie]

We upgraded from 30.x to 3.1.16 last night. Doesn't appear to have resolved the issue so will continue to concentrate on any perceived

cabling issues. As this is now only affecting 3 out of 100 APs it's not as bad as others seem to have been affected.

[Macguy]

Bleeding edge switch there LOL.  We had a few back in the day.  That test pretty much confirms it is nothing on your end.  I think we have two issues here in general.  Last Friday, I had an MR56 "ethernet speed degraded" connected to an Aruba 6405 v2 chassis.  The connectivity history just started to show periods of no connectivity and connected gateway.  Did a cable test from switch and it passed even though statistics show a high amount of CRC errors.  I went over to school and noticed the patch cable from biscuit box in ceiling to the AP was damaged/crimped.  After replacing, there have been no issues this week. 

I do feel however that there are issues with firmware causing AP's to connect at 100 Mbps.  I've had dozens of AP's (various models) trigger this alert.  I'm running firmware 30.7 at each of my 12 buildings.  I believe it started with the 30.x firmware.  The problem is that the 30.x firmware fixes several issues we've seen related to our wifi 6 AP's which is why I haven't rolled back.

[Sladie]

Totally agree with your summary. We feel it has highlighted some cabling issues but also false positives introduced with the FW version. We are also on 30.x so planning to move to 31.x as someone mentioned this fixed the problem so will see if that is the case for us.

[Balcarras]

Update.

The AP is now running at 1000!

I wonder if it did immediately start running at 1000, but it took the portal a while to catch up?

We are now going to try another patch lead to the same AP / Switch and if that works, use those two "known good" patch cables when we put the AP back in to the classroom. If it then drops to 100 we know the issue is with our structured cabling 😞

Maybe an expensive bill coming if some of our cabling has degraded 😞

[MaghM]

Here is the common troubleshooting steps can be done for this situation:

- Perform a cable test on the upstream switch port and ensure there are no breaks in the pairs

- Ensure that the ethernet connection between the switch and the AP is less than 100m in length

- Move the AP to another switch port (confirmed to be working at 1000Mbps) and confirm that the uplink speed is still under 1000Mbps

- Use a new ethernet cable and confirm if the uplink speed is still under 1000Mbps

- If the AP is connected via a patch panel, temporarily connect it directly into the switch with a cable that is under 10m in length

- Run a packet capture on the switch port whilst the AP is physically connected. This will allow us to review the LLDP exchange during link negotiation

- Confirm if you are observing this issue on the latest stable release of MR firmware

If anything else has been noticed, please create a new case and mention all the steps that are already done and checked.

[Balcarras]

Out of interest how old is everyone's structured cabling - some of our areas will be 13 years old.

[Sladie]

We have a site built a couple of years ago with MR46s and no issues. Some of the cabling in our other sites is a lot older than 13 years and running in environments not conducive to good cable management so I'm surprised we don't have more issues!

[cliffatSwift]

An old cable is not at the heart of this issue, unless you still have CAT5 or older cables installed. Then old cables may be your issue.

From a previous poster:

See this on our MR36s running 30.7.1 as well and no changes on our side and no way all the cables are bad for all the APs reporting the issue. Tried shut/no-shut and hardcoding to 1000 but neither worked.  APs com back up at 100Mbps.    No way all of us, all of the sudden, have bad cabling, it is surely an issue with the AP firmware.

Now it may be a “Bug” in Meraki’s firmware, but that hasn’t been proven and Meraki is not 100% sure this is the cause.

At one of our locations, I implemented a temporary work-around and we had 3 MR42 APs go offline. The work-around not only brought the APs back online the APs now have been online for 4 weeks. Our current Meraki firmware v. 31.1.5.1

I’m really surprised that it is taking the Meraki team this long (Issue started at the end of October 2024) to resolve this issue.

[SDM]

I have tested cables in one of our buildings they come back as working fine. I was instructed to up upgrade our MR36 to MR 31.1.6, the current stable version. Did that this weekend, this has not fixed the issue, numerous AP still have the error.

[Balcarras]

We took an AP reporting the "ethernet_negotiation_failure" and plugged it directly in to the switch with a known good 1Gb capable patch cable. It went to 1000Mbps connection and cleared the error. We tested the same with another known patch cable and for a few hours it stayed at 1000. We put the AP back in it's room used the two known good patch cables at the switch and room end, the device came back up at 1000 on Friday. Came in today and it's back running at 100 and the "ethernet_negotiation_failure" alerts gone so you'd look at the dashboard in default state and not know anything was wrong.

See attached. Blue box are AP's in the same switch!

We have many more AP's in a mix of states.

We are on MR 30.6 

As advised by Meraki.

[Nick_TM]

A little update on this after a chat with our Cisco TAM and one of the Cisco/Meraki Technical Solutions Managers.

Basically in October Cisco started rolled out an update to customer tenancies (Meraki Portal) to add the 'Degraded Ethernet Speed' Alert. While they had the data being reported previously they didnt have an alert for the circumstance. This has been rolling through each tenancy and as it hits yours, the alert got enabled and you started getting email about it.

This explains the 'multiple cables failed on same time on the same day', your tenancy got the backend update..

So whereas there are some known bugs related to not negotiating at 1Gb (depending on the AP model) it is likely that other models without this bug are real 100Mbit negotiation issues, and they have advised that 50% of TAC cases are in fact faulty cables and has been confirmed. Basically, its you likely had the problem beforehand and had not noticed the slightly darker green switch port.

My suggestion would be to confirm the models with the known bug via TAC case and if its not those models, do some detailed cable tests (not just a continuity test) and check they are up to scratch on quality.

This nugget of information hasnt been advised clearly and does change the thinking depending on the AP model. We have a mix of both cases so are testing cables and doing RMA's.

HTH 

[Balcarras]

We've had this back from Meraki!

Following investigations, we have narrowed down the issue we are seeing with the "Ethernet uplink speed degraded" alerts on the affected MR APs, which causes the link speed being degraded from original 1000Mbps to 100Mbps and confirmed is associated to a known issue. This issue is already being investigated by our development team internally and we currently have no estimate on when the fix will be implemented but we are working on it. 

We saw the alert pop-up initially on some AP's which were connected to switches which only had 10/100 ports and could never have run at 1000, so I believe there are a few scenarios at play here.

Genuine issues with AP's that were connecting at 1000 that have dropped to 100.

AP's that it's now highlighted had faulty cabling but COULD have negotiated at 1000.

AP's that were correctly negotiating at 100 due to switch limitations, but wrongly alerted as being degraded.

[Spalding]

I have about 200 MR 57 APs throughout the environment on different networks and I currently have 6 APs in the ALERTING mode showing 100Mbps. I have reset them to factory, reviewed the switch ports (Aruba 6300M, 6200F & 2930) and there are no issues with the ports and I see no CRC errors or anything to indicate a cable issue. I have taken a spare AP and swapped it out with an alerting AP and I had no issues so if it was a cable, the error would have returned. I have bounced them several times and nothing is working. Running 30.7.1 so the code isn't that old. The only thing I haven't done is switch the Ethernet port on the AP itself, using port 0 as opposed to port 1. Meraki is sending me new APs to replace them. Frustrating they can't find the issue.

[Balcarras]

Interesting that they don't believe a future firmware will fix it?

Who do we apply to to get new AP's shipped out? 😄

[Spalding]

Hello Balcarras,

Meraki never suggested a firmware upgrade, which they usually do so I was surprised. They recommended the factory reset, which didn't work. I just opened a ticket with Meraki and after looking at the issue they said they would just replace them.