This looks to me more like an industrial PC with 4 graphic outputs (or a board like COM Express, PC104 etc) + your board(s) + displays. I would not be developing my own motherboard for this - that would be unnecessary expensive, risky and would take a lot of time.
Not possible to use standard ITX because proprietary chips and connectors.
What you may need is an adapter board which will have input ports like DVI, HDMI, DP and output connectors what you need. Possibly you could design a baseboard for computer on module (such COM Express).
If you are worried about EMC/EMI (I would be), maybe make number of these adapter boards (each with different interface e.g. HDMI to your special connectors, DVI to your special connectors, DP to your special connectors - depends what your industrial PC will have) and place these adapters close to the display - so for longer distance you use the standard and certified cable and interface (e.g. hdmi) and just very close to the display you use your adapter board.
Important is what your goal is:
- if to build as quickly and simply a system which has 4 monitors and can run 2 of them at the same time,
- or if your goal is to build a fully custom solution with four monitors where 2 can run at the same time, the system is neat, can not be simply copied, but cost a lot of money and can take a lot of time to develop (+ more risky to develop than the first option).