Restrict multiple instances of an application in java
There is a library called jUnique which does that and will save you the bother of implementing it yourself.
If you deploy with Java WebStart the SingleInstanceService does this.
See http://download.oracle.com/javase/6/docs/technotes/guides/javaws/developersguide/faq.html#218