Does mailx send mail using an SMTP relay or does it directly connect to the target SMTP server?

mailx can use SMTP. It's configure file is ~/.mailrc

One example is mailx using Gmail's SMTP.

The configure can even be in one command:

mailx -v -s "$EMAIL_SUBJECT" \
-S smtp-use-starttls \
-S ssl-verify=ignore \
-S smtp-auth=login \
-S smtp=smtp://smtp.gmail.com:587 \
-S from="$FROM_EMAIL_ADDRESS($FRIENDLY_NAME)" \
-S smtp-auth-user=$FROM_EMAIL_ADDRESS \
-S smtp-auth-password=$EMAIL_ACCOUNT_PASSWORD \
-S ssl-verify=ignore \
-S nss-config-dir=~/.mozilla/firefox/xxxxxxxx.default/ \
$TO_EMAIL_ADDRESS

If a normal SMTP server is used, it is much easier (see a detailed introduction here):

mailx -v -s "$EMAIL_SUBJECT" \
-S smtp=smtp://smtp.example.com
-S from="$FROM_EMAIL_ADDRESS($FRIENDLY_NAME)" \
$TO_EMAIL_ADDRESS

You can also put these into mailx's configuration file ~/.mailrc

Traditionally, Unix mail and derivatives (and many other Unix tools) use the /usr/bin/sendmail interface, provided by almost all mail transfer agents (MTAs – postfix, exim, courier, and of course sendmail).

That is, the mail program doesn't speak any network protocol – it feeds the message to sendmail via stdin, and lets it handle actual delivery. (This goes back to the days when some mail used SMTP, some used UUCP, some used BITNET...)

Once a message is queued through sendmail, the MTA handles actual message transmission, whether through SMTP or something else. Depending on configuration, it may either connect directly to the destination MTA, or relay mail through another host (also called a smarthost).

Direct connection is more common on servers; relay via smarthost is more common on personal computers on home connections – relaying through your Gmail or ISP/work email account is essential to avoid the blanket "dynamic IP" anti-spam filters.

(Some MTAs such as esmtp or nullmailer are built specifically for home users and always use a relayhost. These don't support receiving mail and are a lot lighter on resources.)

mailx → [/usr/bin/sendmail] → local MTA queue → [SMTP] → recipient MTA → recipient inbox

mailx → [/usr/bin/sendmail] → local MTA queue → [SMTP] → Gmail or ISP/work servers → [SMTP] → recipient MTA → recipient inbox

---

Other programs, mostly the user-friendly graphical clients such as Thunderbird or Outlook, always connect directly to a relay/smarthost SMTP server (again, usually Gmail or ISP/work SMTP server), which transmits the message on your behalf.

Native SMTP support is present in heirloom-mailx, but not in the traditional bsd-mailx.

app → [SMTP] → Gmail or ISP/work servers → [SMTP] → recipient MTA → recipient inbox

---

The third method – connecting directly to recipient's server – is almost never used, and no MUA supports it. On personal computers, using it would cause your message to get rejected (a lot of spam is sent from infected home user IP addresses).

app → [SMTP] → recipient MTA → caught by the spam filter

From the mailx(1) man page, DESCRIPTION section, String Options subsection:

   smtp   Normally, mailx invokes sendmail(8) directly to  transfer
          messages.  If the smtp variable is set, a SMTP connection
          to the server specified by the value of this variable  is
          used  instead.