I don't think that this is a firewall problem. For me it sounds more like a phone system issue.
You might want to look into the SIP (I hope you are using SIP) messages. The "BYE" message should contain an Q.850 error code, which should give you a hint what the problem is
If the firewall would block the connection between both phones on the sited, then you would have an established call with no audio.
If the firewall would block the connection between the phones and the phone system, then the other phone would not even ring.
"The line just goes down at the same time on both ends" when does this exactly happen?
As soon as the other person is picking up the call?
If yes: Then it's probably a codec issue. e.g. Phone A can only use G711, Phone B only G722 -> call gets dropped
If you are using CUCM: Check also the Region/Location settings. Maybe the configured path is not possible here