How does the server find out what client port to send to?

It's part of the TCP (or UDP, etc.) header, in the packet. So the server finds out because the client tells it. This is similar to how it finds out the client's IP address (which is part of the IP header).

E.g., every TCP packet includes an IP header (with source IP, destination IP, and protocol [TCP], at least). Then there is a TCP header (with source and destination port, plus more).

When the kernel receives a SYN packet (the start of a TCP connection) with a remote IP of 10.11.12.13 (in the IP header) and a remote port of 12345 (in the TCP header), it then knows the remote IP and port. It sends back a SYN|ACK. If it gets an ACK back, the listen call returns a new socket, set up for that connection.

A TCP socket is uniquely identified by the four values (remote IP, local IP, remote port, local port). You can have multiple connections/sockets, as long as at least one of those differs.

Typically, the local port and local IP will be the same for all connections to a server process (e.g. all connections to sshd will be on local-ip:22). If one remote machine makes multiple connections, each one will use a different remote port. So everything but the remote port will be the same, but that's fine—only one of the four has to differ.

You can use, e.g., wirehsark to see the packet, and it'll label all the data for you. Here is the source port highlighted (notice it highlighted in the decoded packet, as well as the hex dump at the bottom):

The "connect request (the client program's connect() system call, typically) causes a three-way handshake. The first packet of the 3-way handshake (from client to server) has the SYN flag set, and includes the TCP port number the client program's kernel assigns to it.

You can see this in an article on Nmap vs Natural SYN packets. The Nmap SYN packet decoding has the phrase "source.60058 > dest.22". The "legitimate SYN packet" decoding has the phrase "source.35970 > dest.80" in it. The two SYN packets tell the remote kernel that the packets are from TCP port 60058 and port 35970, respectively.