That's going to be a pfsense question ...
We've done these kinds of configs using StrongSwan but they don't tend to be reliable. You'll test it 5 times and it will work 4 times. Then the one time you actually need it to work it doesn't.
One thing that might help if you use 15.x code is to use IKEv2, and specify a peer identity for pfSense to match on (instead of matching on your public IP address).