The MX does not participate in RTSP. It mearly masses the packets on. Your configuration will make RSTP see a 10Gb port appear to be plugged into 1 Gbe port. I can see why it would get upset. Only single connect the MXs to the two stacks, using only the 10Gbe cables.
... View more