There are three ways to transmit audio and video over CAT5 or UTP cabling. One is sending audio and video signals using the UTP cable for the physical cabling, commonly called audio and video over UTP. In that case there is no 'protocol', you are simply using the UTP cabling as the physical cabling path for the audio and video signals. Another is using proprietary protocols such as Cobranet, EtherSound, Dante, etc. The third option is streaming, in which case the audio and video is converted to data and distributed using Ethernet. The switches, routers, hubs, etc. that can be used depend on the approach being used.
For example, other than the cable, connectors and possibly any physical patching, you cannot run Ethernet over audio/video over UTP equipment and you cannot run audio and video through Ethernet switches, routers, hubs, etc.