Meraki 17.2.1 Firmware Axis Camera Issues has anyone else experienced this?

[jayj20]

I was looking a bit on where to go for an issue we had with the latest 17.2.1 firmware update. After the update to our network was complete. Not all POE devices,  but specifically axis cameras would intermittently just stop transmitting data across the switch, we would observe the port for traffic but there would be zero and the camera would be unreachable. POE was showing it was still powered and the device would show link  and arp on the switch, but the camera would not ping and be unreachable. This caused issues on our network as every day we would have random cameras just dropping, they'd stop communicating and the only way to get them back would be to cycle the port but then it was just a matter of time before they dropped again. We tried power cycling switches, cameras, and also removing and adding the layer 3 vlan the cameras were on and the issue would still occur.

Finally, we decided to rollback the firmware and just like that the issue had stopped. I had a ticket going but was told to just reach out when we upgrade again for assistance, but I really want to know what the root cause was before we decide to upgrade again. Has anyone else experienced anything similar with the new firmware update?

[RWelch]

What is the switch model being used?  Have you looked at MS 17.2.1 release notes (aka changelog) for the switch you are using in the equation?

[jayj20]

We have a mixed environment of switches MS125-48FP,MS125-48LP, MS225-48, MS225-48FP. It appeared to be happening across all of our meraki switch models. We have another switch vendor in our network and the cameras for these switches never had any issues during this time. We identified it was just the meraki switches. I checked the release notes for known issues and didn't see anything regarding it.

[RWelch]

What does your PoE Usage (budget) look like on each of the switches?  Are they close to being maxed out?  As a test, have you tried to move the axis camera to another port to see if that might make any difference?  Say one that is (was) working fine without an issues or problems?

[jayj20]

The switches did not show to be over budget or denying power to any of the cameras. That was the strange thing, the device would show on arp table, and also the switch would have the POE symbol on it to show that it was currently powered, it just would not be talking to anything. Like it was some kind of frozen somehow, had power, had link light, just zero talking to anything data usage would show zero. We tried moving to another port, but it was happening to almost every camera on our network which was a few hundred, so we were beyond considering it a port issue. And it didn't seem to be happening with any other POE device we had besides the cameras so we ruled out POE

[alemabrahao]

It's probably a firmware bug, but just out of curiosity, what model is it and are you running the most up-to-date version on your Axis cameras?

[jayj20]

We have a few hundred axis Cameras varying in model, Q6055-E, M3066-V, M4036-V, M5525-E, M3006, and a few more. They're all kept up-to-date within our Station Client.

I've been leaning towards a bug I just don't know where to go from here, I don't want to risk upgrading the firmware again without a root cause or no real way to resolve it besides power cycling and waiting for it to happen again.

[alemabrahao]

In fact, the best option I see is to update the switch and perform a packet capture at the time the problem is occurring. You will hardly be able to identify the root cause without reproducing the problem again.

[cmr]

As and when a newer firmware version is released, if you use the staged upgrades option, you can put the least impactful switch in its own group and set the upgrade to do that switch and then many days later the others.  If you need more time you can reschedule the later ones to even later.

[jayj20]

This is what we plan to do from here honestly staged upgrade so we can isolate it. There's no predicting when and where the problem occurs so it's been hard to catch it in the act because we don't know what camera will drop and when.

[PhilipDAth]

Do you have any LACP channels?  You need to delete and re-create them under 17.2.1.

[RJC]

I have this same issue affecting CW9162l WAPS, some (2 or 3 out of the 19 on this network) will not return to normal operation after a port cycle event. No incoming packets. The WAP will commence the boot process, last for 60 seconds then power off. Port on the MS130-24X running 17.2.1 stays up, drawing 5 watts (which is lower than normal operation), but the device  (WAP) doest reach operating state.

I have an open ticket, awaiting a response.

[jayj20]

What ended up being their response did you find out anything more?

[mjheaberlin]

This hit my company yesterday. Meraki Support acknowledged it's a bug, but they don't have a bug ID or known triggers yet. Looks like it may be isolated to only MS130s, but I'm not 100% certain. We have a handful of switches where this problem exists, they recovered one remotely somehow even though they say part of the workaround/fix is a physical power cycle. I would say stay away from 17.2.1 for the time being. They're recommending for us to use 16.9.

Potential workaround/fix that was provided to us:

1. Disabling PoE ports
2. Power off switch
3. Configure firmware downgrade to MS16
4. Power on switch after 10 secs
5. Wait for the switch upgrade and upgrade back to MS17.2.1
6. Re-enable PoE port

[alemabrahao]

Finally they admitted it. 😅

[jayj20]

Is it affecting all of your POE devices or just specific vendors/models?

[mjheaberlin]

All PoE. Cameras, phones, Meraki APs...

[RJC]

Cisco tech reviewed the logs and witnesses the behaviour in real time. Rolled back one switch to 16.9 no improvement.  They’ve issued an rma for my ms130x. Power cycling the switch alone didn’t restore the switch ports. 

[PatTheCat76]

they stop tarnsmitting maybe because they're losing power.

we had similar issue, 2 ap went dark out of 10.

opened a case with cisco they says it's a bug.

they said on rare situations this can happen it's disabling POE. reboot, nothing not even firmware rollback fixed it we ended up having to RMA the units! now eveything works.