Is there a way to know if a Java program was started from the command line or from a jar file?

The straight forward answer is that you cannot tell how the JVM was launched.

But for the example use-case in your question, you don't really need to know how the JVM was launched. What you really need to know is whether the user will see a message written to the console. And the way to do that would be something like this:

if (!file.exists()) {
    Console console = System.console();
    if (console != null) {
        console.format("File doesn't exists%n");
    } else if (!GraphicsEnvironment.isHeadless()) {
        JOptionPane.showMessage(null, "File doesn't exists");
    } else {
        // Put it in the log
    }
 }

The javadoc for Console, while not water tight, strongly hints that a Console object (if it exists) writes to a console and cannot be redirected.

Thanks @Stephen Denne for the !GraphicsEnvironment.isHeadless() tip.

I'm not clear on the question but I'm going to interpret it as you want to differentiate between the following 2

java -jar fred.jar

and

java package.Main

Here is an outline line of the program

import sun.jvmstat.monitor.*;
...
HostIdentifier hostId = new HostIdentifier("localhost");
MonitoredHost monitoredHost = MonitoredHost.getMonitoredHost(hostId);
Set jvms = monitoredHost.activeVms();
for (Object i: jvms) {
   VmIdentifier id = new VmIdentifier("//" + i + "?mode=r");
   MonitoredVm vm = monitoredHost.getMonitoredVm(id, 0);   
   System.out.println(i);
   System.out.println("\t main class: " + MonitoredVmUtil.mainClass(vm, false));
   System.out.println("\t main args: " + MonitoredVmUtil.mainArgs(vm));
   System.out.println("\t jvmArgs: " + MonitoredVmUtil.jvmArgs(vm));
   monitoredHost.detach(vm);
}

The call MonitoredVmUtil.mainClass(vm, false) will either return 'jar' or the name of your main class eg Main.

You have to use $JAVA_HOME/lib/tools.jar to compile and run.

The System.console() trick seems to do the work.

Here's an alternative: there's a method in the class Class getProtectionDomain() which may be used to know the source of the code the the location from there.

The funny is, this method is available since 1.2

I knew I used this before, here's the original answer by erickson

Here's the proof of concept:

public class FromJar {
    public static void main( String [] args ) {
        if ( FromJar.class
                 .getProtectionDomain()
                 .getCodeSource()
                 .getLocation()
                 .getFile()
                 .endsWith(".jar") ) {

            javax.swing.JOptionPane.showMessageDialog( null, "Launched from Jar" );

       } else {
            System.out.println("Launched NOT from Jar :P ");
       }
    }
}

Here's a short ( 1m aprox ) video to see this code running ( and being written with cat :-o )