Plot a legend outside of the plotting area in base graphics?

Maybe what you need is par(xpd=TRUE) to enable things to be drawn outside the plot region. So if you do the main plot with bty='L' you'll have some space on the right for a legend. Normally this would get clipped to the plot region, but do par(xpd=TRUE) and with a bit of adjustment you can get a legend as far right as it can go:

 set.seed(1) # just to get the same random numbers
 par(xpd=FALSE) # this is usually the default

 plot(1:3, rnorm(3), pch = 1, lty = 1, type = "o", ylim=c(-2,2), bty='L')
 # this legend gets clipped:
 legend(2.8,0,c("group A", "group B"), pch = c(1,2), lty = c(1,2))

 # so turn off clipping:
 par(xpd=TRUE)
 legend(2.8,-1,c("group A", "group B"), pch = c(1,2), lty = c(1,2))

No one has mentioned using negative inset values for legend. Here is an example, where the legend is to the right of the plot, aligned to the top (using keyword "topright").

# Random data to plot:
A <- data.frame(x=rnorm(100, 20, 2), y=rnorm(100, 20, 2))
B <- data.frame(x=rnorm(100, 21, 1), y=rnorm(100, 21, 1))

# Add extra space to right of plot area; change clipping to figure
par(mar=c(5.1, 4.1, 4.1, 8.1), xpd=TRUE)

# Plot both groups
plot(y ~ x, A, ylim=range(c(A$y, B$y)), xlim=range(c(A$x, B$x)), pch=1,
               main="Scatter plot of two groups")
points(y ~ x, B, pch=3)

# Add legend to top right, outside plot region
legend("topright", inset=c(-0.2,0), legend=c("A","B"), pch=c(1,3), title="Group")

The first value of inset=c(-0.2,0) might need adjusting based on the width of the legend.