We tried to do the same thing, and ended up just splitting out the subnet and making the IPs in question a /30 and then just disabling the VPN on it.
There's probably a better way to do it, and your solution is probably just missing some NAT configuration, but we gave up and just did it the easy way.