Are you running your cameras on the lowest quality setting possible (720p)?
I'm surprised the Raspberry Pi has enough GPU punch to do 4 video streams, so that is impressive.
This is the GPU requirements for monitoring video streams (note link is broken "right now" but it is being fixed up):
https://documentation.meraki.com/MV/Advanced_Configuration/Hardware_Guidelines_for_MV_Video_Wall_Wor...
I think your fundamental problem is the hardware you are using is not powerful enough. You could use more raspberry pi's.
I tend to use Lenovo P330's which include a Nvidia P1000. I find that it can handle up to 16 cameras just fine, and you can plug up to 4 monitors into it.