What is the proper way of closing and cleaning up a Socket connection?

Closing socket closes the connection, and Close is a wrapper-method around Dispose, so generally

socket.Shutdown(SocketShutdown.Both);
socket.Close();

should be enough. Some might argue, that Close implementation might change one day (so it no longer calls Dispose), and you should call Dispose manually after calling Close, but i doubt thats gonna happen, personally :)

Alternatively, consider using using (yeh):

using (var socket = new Socket(...))
{
    ....
    socket.Shutdown(SocketShutdown.Both);
    socket.Close();
}

According to MSDN, for connection-oriented sockets, if you call Connect, you should call Close. This applies to BeginConnect and ConnectAsync as well.

using (var socket = new Socket(...)) {
    socket.Connect(host);
    ...
    socket.Shutdown(SocketShutdown.Both);
    socket.Close();
}