RGBHV or VGA certainly can support HD signals. VGA is actually a form of RGBHV and I think that what you are referencing is using bundled five wire coax in lieu of VGA with HD15 to five wire BNC breakout adapters at both ends. That will typically reduce the cable losses but the reasons this is not more common probably include that higher resolution HD signals relate to larger bandwidth which in turn relates to greater cable losses and thus more limited practical cable lengths. Another common reason is that many sources will not output high resolution (HD) analog signals or at least not for any protected content.
That brings up the HDCP factor. Will your applications potentially involve encrypted/HDCP protected content? If so then that could be a major factor and may eliminate any analog and SDI format from consideration as those do not support HDCP.
Having just been through a really bad experience with HDMI-to-HD-SDI converters I am wary of relying too much on conversion between those two formats, especially where the sources may be varied and not within your control. Between EDID issues, compatible resolutions and scan/refresh rates, color spaces, etc. I would not count on such converters supporting people being able to connect a variety of personal laptops, tablets, etc. without some form of scaler with flexible input support in the signal chain.
And that segues into another possible factor, what outputs or connectivity may you have to support? Is it realistic to assume all of the possible sources to be HDMI or SDI? Would it make sense to have some form of switcher/scaler that will accept VGA, DVI, HDMI, etc. and convert them all to a single format, single resolution output?
FWIW, many of the video over twisted pair (UTP/STP/CAT) devices are not creating a proprietary signal they are simply adapting the physical wiring to use twisted pair wiring. How they do that may vary and thus prevent different brands or models from being directly compatible but it is not really a proprietary signal.
