Is there a way to run short bits of Java code without compiling?

Yep, you can use Eclipse, create a single project, and create a Scrapbook Page in that project.

You can also specify import statements: http://www.informit.com/articles/article.aspx?p=31789&seqNum=3

Scrapbook pages get their classpath from the containing project's build path. If in a scrapbook page you want to reference a Java element that is not on the build path of the containing Java project, you need to add to the Java project's build path. Scrapbook pages also allow you to specify import statements. You do this by selecting Set Imports from the context menu of a scrapbook page or Set Import Declarations for Running Code from the toolbar. You need to set import statements for references to Java declarations in your projects. This is a common oversight. If the type or package you are attempting to import is not listed in the Add dialog, it means you need to add it to the build path of the project containing the scrapbook page. If you are referencing an element that has multiple declarations, you will need to add an import statement to uniquely identify the element.

Edit: Got another solution too: http://ideone.com. It's an online IDE and debugging tool. You can see an example here: http://ideone.com/98sA8, but it looks like you have to set up a bit more than on a scrapbook page.

Edit 2:

Nowadays in Java 11, if it's a simple app in a single file you can run it directly from the java command (on the command line) which will handle all the compilation for you behind the scenes:

java HelloWorld.java

This is useful for students, as they can get started with Java without learning all of the javac compilation routine.

As of Java 11 (JEP 330) it is now possible to run Java files directly with the java tool:

java Factorial.java 3 4 5

is informally equivalent to

javac -d <memory> Factorial.java
java -cp <memory> Factorial 3 4 5

Java also added support for "shebang" files.

For more details see: http://openjdk.java.net/jeps/330

Using Eclipse, you can create a Scrapbook page which will allow you to do exactly this.

Caveats:

1. You need to use full binary class names for anything outside of java.lang
2. You need to select the code (standard text selection) that you want to run
3. There are three different methods for running -- Inspect, Display, and Run. Inspect and Display are virtually the same, showing you the result of the last statement in your code (so you don't need to print it or anything), Run runs the selected code and dumps any output to the console view.

Never used it, but BeanShell seems to do what you want