How to allow running only one instance of a Java program at a time?

I think that the better idea would be to use file lock (quite an old idea :) ). Since Java 1.4 a new I/O library was introduced, that allows file locking.

Once the application starts it tries to acquire lock on a file (or create it if does not exist), when the application exits the lock is relased. If application cannot acquire a lock, it quits.

The example how to do file locking is for example in Java Developers Almanac.

If you want to use file locking in Java Web Start application or an applet you need to sing the application or the applet.

As the question states that WebStart is being used, the obvious solution is to use javax.jnlp.SingleInstanceService.

This service is available in 1.5. Note that 1.5 is currently most of the way through its End Of Service Life period. Get with Java SE 6!

You can use JUnique library. It provides support for running single-instance java application and is open-source.

http://www.sauronsoftware.it/projects/junique/

See also my full answer at How to implement a single instance Java application?

I think your suggestion of opening a port to listen when you start your application is the best idea.

It's very easy to do and you don't need to worry about cleaning it up when you close your application. For example, if you write to a file but someone then kills the processes using Task Manager the file won't get deleted.

Also, if I remember correctly there is no easy way of getting the PID of a Java process from inside the JVM so don't try and formulate a solution using PIDs.

Something like this should do the trick:

private static final int PORT = 9999;
private static ServerSocket socket;    

private static void checkIfRunning() {
  try {
    //Bind to localhost adapter with a zero connection queue 
    socket = new ServerSocket(PORT,0,InetAddress.getByAddress(new byte[] {127,0,0,1}));
  }
  catch (BindException e) {
    System.err.println("Already running.");
    System.exit(1);
  }
  catch (IOException e) {
    System.err.println("Unexpected error.");
    e.printStackTrace();
    System.exit(2);
  }
}

This sample code explicitly binds to 127.0.0.1 which should avoid any firewall warnings, as any traffic on this address must be from the local system.

When picking a port try to avoid one mentioned in the list of Well Known Ports. You should ideally make the port used configurable in a file or via a command line switch in case of conflicts.