Not out of the box you can’t. When you enable the Client VPN it creates hidden inbound firewall rules that allow access from all IP addresses to the relevant ports on the MX. If you’re using the L2TP Client VPN then it uses UDP ports 500 and 4500, if you’re using AnyConnect then it’s TCP port 443 by default (but you can change this).
I know that support can (in certain circumstances - e.g. for no-NAT) enable manual control of the inbound firewall on the MX, which would potentially then allow you more control over the IP addresses that can access the MX. But that then comes with the overhead that you have to manage all the inbound firewall rules, e.g. ICMP, port forwards, static NATs, etc. so you really need to know what you are doing.
So you might be able to do it, but you’ll need to get support to agree to your request and enable it.
