How can I monitor serial port traffic?

socat is a tool to connect (nearly) everything to (nearly) everything, and tee can duplicate streams.
In your usecase you could connect your serial port /dev/ttyS0 to a PTY /tmp/ttyV0, then point your application to the PTY, and have socat tee out Input and Output somewhere for you to observe.

Googling "socat serial port pty tee debug" will point you to several "standard procedure" examples, one being:

socat /dev/ttyS0,raw,echo=0 \
SYSTEM:'tee in.txt |socat - "PTY,link=/tmp/ttyV0,raw,echo=0,waitslave" |tee out.txt'

The files in.txt and out.txt will then contain the captured data.

Updated after comments:

• The socat syntax looks confusing at first, but in fact, it's just 2 nested statements.
  A small price to pay to such a powerful, versatile tool.
• If you need to setup your serial port, or send other ioctls, do so before calling socat, as socat cannot proxy them.
• The single-purpose-tool interceptty from 2006 has slightly simpler syntax, but can only intercept TTYs (while proxying ioctls), and is probably not in your package manager. (Most Linux distros never added it to their repos)

I don't think the serial driver has any tracing functionality that would allow you to watch packets. You can use strace to observe all the reads and writes from your application:

strace -s9999 -o myapp.strace -eread,write,ioctl ./myapp

I found projects called Linux Serial Sniffer, jpnevulator, and Moni. The first two look like they do exactly what you want. The last one calls itself a monitor, but it actually looks like a standard serial communication program.