Package your panels in a stack-able rack with a back panel. Build all the adapters into this panel, including any matrix and expansion connections you need to make to the panel.
The extra back panel for connections is a great idea and could add to the ruggedness and ease of installation for the panels, as well as protecting the original connectors.
Improvements over the standard panel I/O could be:
Locking power connector, perhaps with loop through (Powercon?)
Locking Ethercon rather than RJ45 for matrix and expansion
XLRs for balanced audio I/O
It should be easy to build in a PSU for a littlite or two to be powered from the rack
Having a UPS with each panel could simplify mains distribution and reduce the single point of failure of one larger UPS covering many panels although it may be easier to monitor the status of the UPS with one larger device. It would be lots of logic programming to trigger alerts for each panel in a large system.
I guess a 3U slanted desktop rack would allow for a 2u panel plus 1u expansion panel if needed, or 2u panel plus timecode display as required.