Pass USB 3.0 connection through D-SUB connector

I assume this is due to insufficient shielding to get a USB 3.0 link?

It's more likely that you simply introduce an impedance break so significant by separating the conductor pairs of the superspeed conductors that communication can't properly take place.

What is the most likely cause for this failure, and how to avoid it in the future, if possible?

As said, you can't just separate the conductors of USB3 arbitrarily: the signal is carried as electromagnetic field between the conductors; because the signal frequencies of USB3 are solidly within the microwave range, your splitting of conductor pair essentially means you break the transport of energy.

You probably won't be able to solve this situation using your current D-SUB connectors at all. You'll need to replace these connectors with ones that at least approximately retain the nominal 90Ω impedance of USB3 SS connector pairs. It's pretty likely the easiest way to achieve that is through USB3 connectors themselves.

There are two problems here:

1) A USB cable is hot swappable, meaning the power pins are engaged first (they are longer than the data pins). A d-sub connector is not built for hot plugging, the pins are all the same length and some engage first depending on the angle that the D-sub is plugged in and is potentially causing problems

2) The impedance of the differential lines (as previously mentioned) needs to be 90Ω, otherwise the fast differential signals will reflect and attenuate if not properly matched. The connector also needs to be impedance controlled to pass fast signals through it.

The USB 3.0 spec uses low-frequency signaling (10-50 MHz) to initiate a link with the other side. SFP+ transceivers usually don’t cover this range, at least not in their datasheets (it’s more like 300-2500 MHz or so). So this vital signal may not reach the other side properly, and hence the link establishment may fail. Source: http://billauer.co.il/blog/2015/12/usb-superspeed-parallel/

The problem with your d-sub is it probably has a capacitance\inductance simmilar to the ones shown below, and not fast enough to pass the fast 2.5GHz signaling of USB 3.0

Source: https://www.farnell.com/datasheets/66098.pdf

So what can you do about it?

If you don't have to have USB 3.0, you may want to try using only the GND, Vcc, D+ and D- lines. If hot plugging of the D-sub can be avoided that might be best (plug in the connector that goes to the hub).

They also make UHV compatible USB 3.0 port feedthroughs if you want to drill another hole in your plate or find another way to pipe it in.

The problem with DB-9 connector is that it is not "impedance controlled" and has no shield between signal pairs (you need to use differential pairs through the connector and shield them from other diff pairs). USB 3.0 operates at 2.5 GHz signal rate, and "3 cm loose wire" is a kill for it. Impedance mismatch creates multitude of signal reflections (causing so-called "inter-symbol interference"), and significant cross-talk between Rx and Tx pairs will kill signal coherency causing massive link drops. USB 3.x specifications have very strngent requirements for impedance and near- and far-end crosstalks over the cable.

To have USB 3.x connection with your internal camera, you need either to do a heavy search for vacuum-grade USB connectors (if they exists, they might), or use RF-grade 50-Ω coaxial multi-pin feed-through connectors, space-grade. There are twin-axial Sub-D size connectors, you need at least two twinaxial channels, similar to this one:

You also could use connectors used for ethernet connectivity, if you can find them in vacuum grade version, something like this

In worst case you can use four feed-through SMA-type connectors, and make a USB-to-SMA adapter, similar to what USB-IF uses in interconnect testing and cable certification, something like this:

Some sources for vacuum-grade feed-through solutions are Pave Technology, MDC Vacuum Products, and likely many others.

In any case I see no chance to have a reliable USB SuperSpeed channel using DB-9 connectors, and you will need a serious rework on your chamber.