As already mentioned, it will not break your incoming connections. The load balancing works strictly outbound, while your port-forwarding works inbound. But with two ISPs that are also quite similar, you could also think about providing inbound access through both ISP which gives you additional redundancy in case the first ISP fails.
If you found this post helpful, please give it Kudos. If my answer solves your problem, please click Accept as Solution so others can benefit from it.