30.7 Update Breaks RADIUS

[jpeterprarch]

We just recently upgraded our eligible APs to 30.7 from 30.5.  Immediately afterward all 802.11 auth attempts to our RADIUS servers failed.  Thus, we rolled back to 30.5.  Has anyone else encountered this issue?  I see in the 30.7 release notes a reference to a fix for RADIUS requests when if they had exited the primary management interface if an AMI was defined.  In our environment we don't have a AMI defined.

[Brash]

What are you using for your RADIUS server?

We have upgraded our fleet to 30.7 and are using RADIUS authentication without issue.

[jpeterprarch]

Windows NPS

[ammahend]

I have 30.7 in production and a test lab on 31.1.2 no issues with radius, I am using Cisco ISE as radius.

can you share more information like, if you are seeing logs on radius and share the logs to understand why radius authentication is failing.

[RaphaelL]

Exact same setup as @ammahend , running without any issues. 

WPA2-Enterprise , Cisco ISE reachable via AutoVPN.

[jpeterprarch]

We're using Windows NPS.  The requests never hit the NPS servers therefore there's nothing to see in the logs.  

[ammahend]

out of curiosity re-enter shared secret again on both side and see if it changes anything.

[Mark_S]

Hi jpeterprarch,

There is an issue in the 30.7 firmware, under very specific configurations, whereby client balancing rejects clients. This may be what you were seeing when you upgraded.

If you have not already, I would recommend raising a case with Meraki support so they can assist with troubleshooting this issue to help determine whether you were indeed affected by this.
Details on how to raise a case through dashboard or to call the support line can be found within the 'Get help & cases' page which is accessible in the ? icon menu in dashboard.

[jpeterprarch]

Ah, thanks.  I have opened a case and waiting to hear back.

[ChristianHenn]

Meraki support said there is no known bug. Difficult to troubleshoot this topic when the manufacture can't confirme the bug.

[jpeterprarch]

Well.  @Meraki.  Look again.  I received a single response from support asking the obvious questions.  Haven't heard back.  

[redsector]

I did open a case but they didn´t really understand the problem.

[skunkytoots]

30.7 broke Radius for my org as well

[Brad39]

Hi ,

Adding to this as well, we have upgraded to 30.7 and have seen two networks with the same issue, generic Association errors with SSIDs with Radius Auth. Running Windows NPS. Will Raise a support ticket to investigate. 

[jpeterprarch]

I'd be curious to hear what you hear from support.  So far I have yet to hear back.

[pjc]

@skunkytoots and @Brad39 do you both have client balancing enabled on the affected networks ?  I've scheduled 30.7 upgrades for next week and we run NPS Radius but we don't use client balancing

Thanks

[jpeterprarch]

Yes we have Client Balancing enabled.  Perhaps we fall under "very specific configurations".

[Brad39]

We have it on, however we tested turning this off to try and fix the issue but didn’t help

[ChristianHenn]

We had the exact same issue. The only solution was to rollback.

[pjc]

I've decided to postpone my planned rollout of 30.7.  Although we don't use client balancing, @Brad39 stating that turning off client balancing didn't solve the issue is enough for me to be nervous about this, despite me running 30.7 in a small live environment (2 x MR44's) for the past week without issues with radius auth (my normal live test firmware update environment)

[Brad39]

It’s a strange one, in the end this looks to have only affected 2 out of around 30 networks in our org, all very similarly configured. We are just going to hold off on 30.7 for those two and possibly jump to the next release.

[redsector]

I have got the same issue

[redsector]

I have got also issues with Cisco ISE 3.2 Errors occured since ugrading ISE from 2.7 to 3.2 and Meraki MR30.6 to MR30.7

[AAABBBCCC]

Had this exact issue going from 29.X to 30.7.  Any SSID using radius was broken.  We backend with ISE.  Rolled back to fix for now.

[user61762]

Similar experience here after seven networks with MR42 (and some MR86) APs upgraded from MR 29.5.1 to MR 30.7 overnight.

We have various SSIDs configured across the seven networks in question.

We experienced at least two networks where an SSID configured for Identity PSK with RADIUS did not permit clients to move past the association step.

Event logs showed clients attempting to associate but never moving to the wpa_auth step. After not too long, associations from clients on the SSID in question were failing with reason code 17.

Finding this thread (and another thread on Reddit), we disabled client balancing and the symptoms subsided.

The remaining networks that have an SSID configured for Identity PSK with RADIUS do not currently have active clients that would be attempting to use that SSID, so we don't yet know if they're affected.

Like others, we've opened a case with Meraki support to report the issue and seek more details.

[pjc]

Being reported on Reddit

https://www.reddit.com/r/meraki/comments/1e35rty/mr_307_causes_association_error_17/

I'm gonna hold of upgrading until confirmed fixed in future release

[jpeterprarch]

This might be it.  Our affected SSID is in Slot9.

[mags1892]

 this seems specific to the AP model , we have MR57 and 30.7 with no issues . However other offices with mr33 do have issues. Meraki have not been very forthcoming with support 

[BWIT]

Hi

We had the issue with MR46's with NPS

One of my networks I applied the workaround of diabling the client balencing, the others I chose to roll back.

The best part is meraki support said

'a fix has been developed by our engineering team and should be available in the next firmware releases of MR 30.8 & MR 31.1.2'

However 30.8 is not yet avaible and 31.1.2 has already been superseeded by 31.1.3 in beta. 🙄

Even worse than that, I asked why this known issue was not noted on the release notes and could the add it, they said

'While it is true that there is a known issue with SSID and client balancing in version 30.7, not all customers utilize these specific features, so they will not be impacted by this issue.' ...'Furthermore, the request to update the release notes of firmware upgrades immediately upon issue identification cannot be fulfilled by Meraki Support directly. Nevertheless, we have the means to relay feedback and information to our internal development team. Notably, utilizing the "leave feedback" function'.👏

The point being, I dont know why they can't retrospectivly add a note the release notes of 30.7 to mention this known issue, its not a secret.

Finally, the best part, even though I have an active support case about this issue, Meraki's automation decided to schedule updates to 30.7 for my remaing networks last week. 😒

[JMI]

As per the recent findings in the Reddit thread mentioned earlier - avoid SSID slot #9.

[jpeterprarch]

Just read it (pun intended).  You have got to be kidding me.  Our affected SSID is in slot 9.  Now I haven't fully tested it yet, however, if this is the reason, this has to be one of the most absurd bugs encountered in my entire 30 year IT career.  I echo the sentiment of another post prior "Doesn't anyone test anything anymore?"

[ChristianHenn]

Our affected SSID is in different templates in different slots...

[JMI]

I now found they actually updated the 30.7 release notes a couple weeks back. Not that it would have made much of a difference to us, we only have three active SSIDs. But at  our main site (of course) one of them was in #9, due to the presence of various disabled legacy networks.  No RADIUS on that SSID, by the way.

One might be forgiven for thinking Meraki could have issued a warning to accounts where they saw this particular configuration, them having access to global inventories and all.

[Brad39]

So looks to be an update in the "Known Issues". 

None of our affected networks are in slot 9 SSID, however they are in Slot 10. Meraki has updated the known issue to now be SSID 10. We are going to look at moving these from that SSID slot to another one in the next week or so and re-test 30.7...

[redsector]

Hello, where can I see the "slot"-number?

[Brad39]

Under the network, Wireless > Configure > SSIDs and then click show all my SSIDs. Left to Right is 1 to 15

[TBHPTL]

No, in GUI left to right it is 0 , 1, 2 ,3 

[TBHPTL]

https://developer.cisco.com/meraki/api-v1/get-network-wireless-ssids/

Left to right in GUI is slot 0 , slot 1, slot 2, etc

[Brad39]

Sorry you are correct!

Pulling it via API it’s “Number” 0-14 and via the GUI “Unconfigured SSID” 1-15. 

The terminology is prob not the best.

Sorry!

[mags1892]

we only have 4 ssids and the radius issues is there . we downgraded firmware and used meraki proxy on the MRs but also had to upgrade firmware on the MX 

[redsector]

Which MX firmware had you bevore and after upgrading?

[mags1892]

MX 18.211.2 - 18.211.3 

MX - 75

[redsector]

Thank you, very interessting. we will check it here, we also have MX75 and firmware MX 18.211.2

[mags1892]

it does seem like this had a bearing whereas our mx100 worked without issues. 

[JMI]

That's weird, so this bug is moving across the configured SSIDs? In our case it was definitely the 9th SSID that was affected (Default name "Unconfigured SSID 8" as they start counting from zero).

[scpswifi]

Same issue here, I was using slot 10 with radius and it broke.  Moved the network to slot 8 and clients are connecting again.