What is the correct way in Rust to create a timeout for a thread or a function?

As @Shepmaster noted: it's a bad idea to terminate threads.

What you can do instead is to give the thread a Sender through which it should notify you if it has successfully opened a connection (maybe even by sending you the handle). Then you can let your main thread sleep for the time you wish to wait. When your thread wakes up, it checks its corresponding Receiver for some sign of life from the thread. In case the thread did not answer, just release it into the wild by dropping the JoinHandle and the Receiver. It's not like it's consuming cpu-time (it's blocked), and it's not consuming too much memory. If it ever unblocks, it'll detect that the Sender is not connected and can shut down for good.

Of course you should not have bazillions of these open threads, because they still use resources (memory and system thread handles), but on a normal system that's not too much of an issue.

Example:

use std::net;
use std::thread;
use std::sync::mpsc;

fn scan_port(host: &str, port: u16) -> bool {
    let host = host.to_string();
    let port = port;
    let (sender, receiver) = mpsc::channel();
    let t = thread::spawn(move || {
        match sender.send(net::TcpStream::connect((host.as_str(), port))) {
            Ok(()) => {}, // everything good
            Err(_) => {}, // we have been released, don't panic
        }
    });

    thread::sleep(std::time::Duration::new(5, 0));

    match receiver.try_recv() {
        Ok(Ok(handle)) => true, // we have a connection
        Ok(Err(_)) => false, // connecting failed
        Err(mpsc::TryRecvError::Empty) => {
            drop(receiver);
            drop(t);
            // connecting took more than 5 seconds
            false
        },
        Err(mpsc::TryRecvError::Disconnected) => unreachable!(),
    }
}

The answer by @ker will always wait 5 seconds, even if the connection finishes more quickly. Here is a similar approach where the timeout and network request both happen on separate threads, and the first one finished wins:

let (sender, receiver) = mpsc::channel();
let tsender = sender.clone();
let t = thread::spawn(move || {
    match sender.send(Ok(net::TcpStream::connect((host.as_str(), port)))) {
        Ok(()) => {}, // everything good
        Err(_) => {}, // we have been released, don't panic
    }
});
let timer = thread::spawn(move || {
  thread::sleep(Duration::from_millis(5000));
  match tsender.send(Err(MyTimeoutError)) {
    Ok(()) => {}, // oops, we timed out
    Err(_) => {}, // great, the request finished already
  }
});
return receiver.recv().unwrap();

But as long as you're doing that, you might as well just use recv_timeout instead:

let (sender, receiver) = mpsc::channel();
let t = thread::spawn(move || {
    match sender.send(net::TcpStream::connect((host.as_str(), port))) {
        Ok(()) => {}, // everything good
        Err(_) => {}, // we have been released, don't panic
    }
});
return receiver.recv_timeout(Duration::from_millis(5000));