using a COM port - Close after each use, or leave always open?

Calling SerialPort.Close() frequently is a mistake. Having another app steal the port away from you isn't exactly very desirable. But more problematic, and the problem you are having, is that Close() doesn't wait for a worker thread that is started by SerialPort to exit. That worker thread raises the DataReceived, PinChanged and ErrorReceived events. It takes "a while" for it to exit, could be between milliseconds and seconds. Calling Open() again will fail until that's done.

It's a flaw in the class, but induced by the common usage for serial ports. Apps don't normally close them until the app terminates. Including never, avoiding a common deadlock scenario. Do note that the MSDN article for Close warns about this:

The best practice for any application is to wait for some amount of time after calling the Close method before attempting to call the Open method, as the port may not be closed instantly.

There is very little harm in leaving a serial port open, so yes, keep it open. It saves you the overhead of open/closing it.

If you're worried about the opening/closing and other apps stealing the COM port away, you could use the approach used by Microsoft for the GPS intermediate driver in windows embedded, i.e. to write an aggregator, one which opens the port, keeps it open, then provides connection points for other apps to connect to.

How you create the connections is up to you: you can get right down deep in the hardware and write a virtual com port driver that's shareable, or you can do what I did and write a simple win32 socket service that allows client programs to connect via regular windows socket connections.

Maybe not a straight forward answer, but food for thought.