There's really no way to do this with the Meraki gear.
Users will always be able to lookup their public IP address by going to a page like Instant IP Address Lookup (whatismyipaddress.com)
The only options that come to mind are:
- Tunnel all traffic to a SaaS gateway/VPN (Cisco Umbrella, ZScaler, Cloudflare etc.) so that their public IP is what will show, not your one.
- ISP configured CG-NAT, in which your outbound IP address will actually be a private IP NAT'd to the ISP's public IP.
That being said, I don't think either of these options are justifiable for that specific purpose alone.