How do I know if a serial port is actually transmitting data, without opening the device?

RS232 has no "cable presence" indicator of any kind. You're just getting transmission or metadata (control) signals through, or you don't - that's all you know. If you receive an incoming signal (CTS|DSR|CD) you know the cable is connected. If you don't receive any incoming signal, the state of the cable is indeterminate and there is no way to determine if it's plugged in without additional hardware solutions - or performing some kind of exchange with the remote device.

The usual approach is performing some kind of "keep-alive" transmissions (even just metadata - e.g. momentarily set DTR and expect CTS) but if the discipline of protocol used by software at the two ends of the cable forbids such idle exchange, you're pretty much stuck with using a soldering iron to proceed.

What you might try, is some kind of additional "demon" that sets up a pipe, forwarding data between your software and the physical device (on both ends), encapsulating it - and performing "connection checks" if the pipe is idle.

Let me add one rather common solution: if your endpoint device doesn't use hardware control, you can short DTR with CTS inside the plug on the host side and use 'hardware control' on the host side. Generating DTR automatically drives CTS, enabling the transmission, if the cable is present, so transmission is unaffected. Meanwhile, with cable absent, the system will react to lack of CTS in a manner appropriate to this event, e.g. generating a timeout or suspending transmission until the cable is plugged in.