How to disable stack trace generation in a java program?

I'm also curious on why you want to do this, but if you really have your reasons, you have at least two options:

If you want to disable stack trace generation for your own Exception implementations, you can simply override the fillInStackTrace method:

public static class MyException extends Exception {
    @Override
    public Throwable fillInStackTrace() {
        return this;
    }       
}

If you want to disable it for all exceptions, you can use a byte code instrumentation agent to replace the fillInStackTrace method in the Throwable class. That will however only work for Java 6, since you in Java 5 are not allowed to replace a native method (fillInStackTrace) with a Java method using instrumentation.

1. I don't think it is possible for code to know that it is being debugged, except by indirect (and unreliable) means like measuring how long it takes to execute code sequences.

2. It is not possible to disable all stacktraces. You can disable stracktraces for exception classes that you define yourself, by overriding Throwable.fillInStackTrace() to do nothing. But that won't work for classes that you cannot change.

But if you are thinking of doing these things to prevent reverse engineering, you would be wasting your time even if you could do it. It would be simple for a hacker to identify your application's anti-reverse-engineering code and edit the relevant bytecode files to disable it.

EDIT - I've revised my opinion on what you are trying to do. Given that what you are doing is distributing an SDK which you are expecting your customers to embed in their own applications, disabling stacktraces for the entire Java application counts as customer hostile behaviour, IMO. As a side-effect of protecting your "precious" IP, you are making it hard for the customer/developer to debug her own code. Even code that doesn't have your precious methods on the call stack!

If I was a customer, I'd probably prefer that you shipped obfuscated code than do this. But most likely, I'd try VERY HARD to find an alternative software supplier that didn't treat its paying customers as thieves.

There are a few intricate parts of the JVM (at least, Sun's implementation of the JVM) which do not work if stack trace generation is disabled (I saw this in the implementation of some support methods for reflection). So I do not think that stack trace generation can be disabled at all. The Runtime.trace*() methods are about something else (a debugging tool much more thorough than stack traces).

In all generality, any Java code can be transparently analyzed, if only through bytecode instrumentation (bytecode modified with extra instructions when it is loaded). The only known defense against such analysis (I am assuming that you are trying to keep your code internals confidential) is obfuscation. See for instance ProGuard. Obfuscation will make stack traces useless to any over-inquisitive user (and, sadly, it also makes debugging very difficult, for the same reasons).