MAC IOS Randomization "Fix" results in 70x drop in Passersby traffic

[Cloud11]

Greetings - We noticed a multiple orders of magnitude drop in "Passersby" traffic counts between Jan.20 (122,420) and Jan.21 (1765). So I asked Meraki support for an explanation which went like this:

Meraki: we implemented a solution to this on January 21st to bypass this randomization and give an accurate representation of your passerby traffic (emphasis mine)

Me: Interesting – how did you bypass MAC randomization for iOS devices that have never authenticated to the WiFi network?

Meraki: For passerby traffic to be generated on the dashboard we don't need devices to connect only for them to send an association to any SSID/AP or the like and our APs see that. As far as what our development team did to workaround that MAC randomization feature, unfortunately, I have no visibility into that code base or actual inter-working of how that change was implemented and even if I did, I would be unable to provide you with it as it would be considered proprietary information.

Me: (mmkay) I still need to explain to my clients why Meraki has been providing misleading data for the last seven years and why this massive change in their analytics is now more accurate considering iOS has developed technology specifically to thwart this kind of data capture.

Still waiting on a response to this, but if anyone has any insight into this I'd be grateful.

Until then I will be telling clients that Meraki's solution was: "WE DIVIDED THAT BIG NUMBER BY 70!!!"

Thank you.

[BrandonS]

This may be helpful?  At least to read their extensive disclaimers around the accuracy of analytics due to MAC randomization. https://meraki.cisco.com/product-collateral/location-analytics-whitepaper/?file

This part kind of admits it is a difficult problem and so difficult to be accurate:

Location data is largely captured per device using that device's media access control (MAC) address as a unique identifier. As part of a privacy technique, some mobile operating systems have added functionality that attempts to randomize the WLAN MAC address a device uses, making it more difficult to track by solutions such as Meraki Location Analytics. As the number of mobile devices that implement randomization increases, solutions to detect and locate devices have changed. Meraki provides additional capabilities such as bluetooth information via the Meraki Scanning API, enabling Meraki customers to anonymously include wearable devices as part of their location analytics dataset.

Based on empirical evidence from Meraki's own experiments and those of our analytics partners. This behavior tends to vary greatly based on the operating system and which apps are installed on the phone for example, if a certain app is very active, it could cause a device that is asleep to probe several times a minute.

[PhilipDAth · Accepted solution]

https://documentation.meraki.com/General_Administration/Cross-Platform_Content/Meraki_and_iOS_14_MAC... 

"You might see a decrease in “Connected” clients as devices transition to randomized MAC addresses because the Meraki dashboard will filter out all randomized MAC addresses by default.

You also might notice a decrease in “Passerby” and “Visitor” because the Meraki dashboard will also filter out all randomized MAC addresses for these categories. Please note Apple devices have been using a randomized MAC address for Probe Requests prior to the iOS 14 release.

Loyalty and Engagement graphs will be impacted in a similar way but will stabilize over time."

[PhilipDAth]

ps. If you are interested in analytics, start looking at transitioning those clients across to MV Analytics instead.

[BrandonS]

Good point, @PhilipDAth 

I am not deep into the analytics industry but it seems this may be where the industry is headed for accurate head counts that are important for retail and COVID occupancy and other things I am sure I am not thinking of.  

Counting bodies is probably better not only because of MAC randomization, but also the fact that many people have multiple devices like smart watches, wearables, etc. that are only going to increase. I think it will just become increasingly difficult to accurately count bodies attached to devices as device counts per person increase and (hopefully) privacy increases as well.

[PhilipDAth]

The other is cost.

If you want to generate heat maps over floor plans, you need each part of the floor plan to be visible by 3 to 4 APs to be able to do triangulation.

You need 1 camera.

[Cloud11]

@PhilipDAthFamiliar with the MV capabilities, but never tried IRL. What's your experience been in terms of accuracy, de-duping, range, etc?

We tend to focus on outdoor shopping centers so heat maps are better for show and tell vs. analysis or deriving meaningful information.

And considering our locations are mostly outdoors, positioning APs is more opportunistic vs. indoor grid deployments so accurate location data is basically unachievable at this point.

Appreciate your input!

[PhilipDAth]

What is IRL?

[BrandonS]

IRL = "in real life", old man..  😉

[PhilipDAth]

Accuracy = good

De-duping = a single camera can track unique humans

Range = Not so great.  I've been using them indoors up to maybe 20m.

[Cloud11]

@BrandonSAgree with you 100%. We advise clients that these analytics should only be used directionally and it is NOT an accounting system. I wouldn't use WiFi as a reliable platform for any kind of Covid tracing/tracking unless you could map MACs to emails or MDNs for communication about proximity. Even then, I'd have a 200 page disclaimer about false positives and accuracy.

[BrandonS]

Interesting thought about COVID proximity tracking.  I was imagining  just complying to 25% capacity type regulations and how that could be difficult when one person = more than one device.  

Anyhow, here is a one minute MV analytics video that you might like to see: https://www.youtube.com/watch?v=MgqbvIvJHcY

[Limitless]

It took 4 years to get them on to wi fi now install cameras instead. Well there's that client gone. MV just reasonable not the answer only in certain use cases.

[BrandonS]

>It took 4 years to get them on to wi fi now install cameras instead. Well there's that client gone. MV just reasonable not the answer only in certain use cases.

I get it, but try to find the upside.  No one expects you can control decisions like this made by the likes of Apple and Google, etc.  Technology is always changing and always will.  It is an opportunity to get more accurate data and for you to make more sales and learn new technology.  And all that while helping protect the privacy of individuals.  If your customer would rather have nothing after a move like this from Google and Apple then all you can do is show them the available options.

Best.

[Limitless]

Brandon, we already use the MV API's so no biggy here for us.  It would be much easier to record the random MAC addresses and allow tech companies to sort this out themselves as we do. Our real opportunity is to serve up offers and suggestions based on location & previous purchasing history. With cameras we cant communicate with the customer, we cant detect abandoned baskets and we cant recommend new products but hey ho we'll get over it.

[WB]

Not a huge shock unfortunately! One of the primary reason for MAC address randomization coming in is to stop things like tracking analytics having a DB of a users movements. At the moment it's just 1x random MAC per-SSID but I'd expect that to become either per-connection or 24 hour rollover by default in the future.

Tough one for retail that previously leveraged those footfall/passerby stats and I'm not sure of how you can get around it!