So I did a few packet captures and what I noticed is that the people on the regional Fiber ISPs had packet fragmentation issues whereas people on Charter/Spectrum and Verizon/ATT did not have fragmentation issues. I tested further and manually set the MTU on the people having issues to 1390 after some ping tests and that resolved the connection issues.
... View more