If you capture your WAN interface and filter on the host IP of the other side or filter on port 500 or port 4500 you have to see the security association frames. It might be possible that you'll have to start a ping from an inside interface on your MX to an IP on the LAN on the other side (even if it does not exist) to trigger an attempt to bring up the tunnel.
Also watch on your fortigate if it is set to responder only that you have to initiate on the MX in that case.
If it is IKEv1 you should see 6 main mode messages and 3 quick mode messages. If you use IKEv2 you only have 4 messages.
You have to filter on the same initiator SPI to get the packets only pertaining to a certain session or setup your wireshark with an IPsec profile containing useful columns to quickly see where in the exchange it is failing.
You will see messages with INFORMATIONAL message coming from the device that does not like what the other side is sending.
Most common are:
- One or both of the devices are behind NAT and you are getting the IKE-ID wrong. (failure will happen right after the packets are encrypted)
- You still have wrong parameters at any end and there are no matching phase 1 policies (you should see it fail after the first message)
- You use IKEv2 and you have multiple local or remote networks and the device on the other side can't put everything in one SA.