How to manually install an artifact in Maven 2?

You need to indicate the groupId, the artifactId and the version for your artifact:

mvn install:install-file \
  -DgroupId=javax.transaction \
  -DartifactId=jta \
  -Dpackaging=jar \
  -Dversion=1.0.1B \
  -Dfile=jta-1.0.1B.jar \
  -DgeneratePom=true

According to maven's Guide to installing 3rd party JARs, the command is:

mvn install:install-file -Dfile=<path-to-file> -DgroupId=<group-id> \
-DartifactId=<artifact-id> -Dversion=<version> -Dpackaging=<packaging>

You need indeed the packaging option. This answers the original question.

Now, in your context, you are fighting with a jar provided by Sun. You should read the Coping with Sun JARs page too. There, you'll learn how to help maven to provide you better information about Sun jars location and how to add Java.net Maven 2 repository which contains jta-1.0.1B.jar. Add this in your settings.xml (not portable) or pom.xml (portable):

  <repositories>
    <repository>
      <id>maven2-repository.dev.java.net</id>
      <name>Java.net Repository for Maven</name>
      <url>http://download.java.net/maven/2/</url>
      <layout>default</layout>
    </repository>
  </repositories>

I had to add packaging, so:

mvn install:install-file \
  -DgroupId=javax.transaction \
  -DartifactId=jta \
  -Dversion=1.0.1B \
  -Dfile=jta-1.0.1B.jar \
  -DgeneratePom=true \
  -Dpackaging=jar