using the signal (composite) from a Videonics MX-1 mixer, going through a video distribution amp, signal is routed to a a RadioShack modulator. the sound is from a matrix output on the Allen & Heath GL2400-32 that runs through an RDL audio distribution amp then to the modulator. from the coaxial output of the modulator we used a Muxlabs balun to run on Cat5.
the Cat5 runs from the rear of the main building to the front where existing Cat5 cables were installed when the youth building was built -- this run is about 120 feet. from there to the youth building is approximately 200 feet, then another 60 feet within the youth building to the TV used for display. another Muxlabs balun changed the signal back to the coaxial (RF) from the Cat5.
I'm not real surprised that the video and audio were poor quality. You took a, by today's standards, low quality video signal and put it through a number of conversions using less than professional equipment. It was then run 380' on CAT5 when MuxLabs seems to recommend a maximum of 330' for Channel 2 and reduced distances or an amp with high frequency compensation for higher channels, which would apparently include the Channel 3/4 options likely supported by your modulator. That would likely result in a poor video image.
On the audio side, you have the same distance issue with the RF over CAT5 run and I'd also guess that the Radio Shack modulator was likely looking for consumer level unbalanced, -10dBV audio inputs while the Matrix Out on your console is an impedance balanced, nominal -2dBu/maximum +20dBu TRS output, so you easily could have been overdriving the modulator audio inputs. All in all, not very surprising that the results were less than acceptable.
My first thought is to cut out the RF portion of the signal path if you can, going direct baseband video and audio over CAT5. MuxLabs and many others offer composite video and audio baluns that support runs much longer than your 380' and that would also remove the RF modulation and demodulation. You are still going to be limited to the quality of the composite video signal out of your MX-1 and may still have to watch the audio levels.