ggplot2, legend on top and margin

Like you said, I can't see it in your example, but I'm guessing the margin is of the legend itself. You can eliminate the margin around the legend itself by adding:

+ theme(legend.margin=margin(t = 0, unit='cm'))

This applies to ggplot2 v2.1.0 or higher. Note that, at least for now, the old solution still works as well:

+ theme(legend.margin=unit(-0.6,"cm")) # version 0.9.x

In the year since this question was asked/answered, ggplot entered maintenance mode, so there won't be any future updates (meaning the OP's strategy of waiting for an update won't work).

The accepted answer relies on fudging the margin around the legend with legend.margin. However, this doesn't generalize well, especially when using ggsave() with different sizes or scale factors. There is fortunately a more generalizable, universal solution, though.

legend.margin only takes a single value for padding on all sides, while plot.margin takes four values for the top, right, bottom, and left margins. The default margins are based on lines (rather than mm or inches), like so: plot.margin=unit(c(c(1, 1, 0.5, 0.5)), units="line")

If you set legend.margin to 0, you can use negative plot.margin values, based on line units, to move the legend to the edge of the plot area. Setting the top margin to -0.5 works perfectly:

ggplot(diamonds, aes(clarity, fill=cut)) + 
  geom_bar() +   
  theme(
    plot.margin=unit(c(-0.5, 1, 0.5, 0.5), units="line"),
    legend.position="top",
    plot.background=element_rect(fill="red"),
    legend.margin=unit(0, "lines")) +
  guides(fill=guide_legend(title.position="top"))

The same idea works if the legend is positioned at the bottom:

ggplot(diamonds, aes(clarity, fill=cut)) + 
  geom_bar() +   
  theme(
    plot.margin=unit(c(1, 1, -0.5, 0.5), units="line"),
    legend.position="bottom",
    plot.background=element_rect(fill="red"),
    legend.margin=unit(0, "lines")) +
  guides(fill=guide_legend(title.position="top"))

As long as you set the margin of interest to -0.5 lines, the extra whitespace should disappear. This should work at any viewport size and any width/height/scale combination with ggsave()

If I exaggerate the margins for more visibility, and run showViewports, I get the following:

p + guides(fill=guide_legend(keyheight=unit(1,"cm"))) + theme(plot.margin=unit(c(1,1,1,1),"cm"))
showViewport(col="black",label=TRUE, newpage=TRUE, leaves=FALSE)

from which it would appear that the non-existent title is somehow taking space.

Edit: nope, it's just an unfortunate overlap of the labels. It's not the title.

Let's look at the legend itself, which seems to be causing the problem.

library(gtable)
g = ggplotGrob(p)
leg = gtable_filter(g, "guide")
plot(leg)
leg$heights
# sum(0.5lines, sum(1.5mm, 10mm, 0mm, 1.5mm), 0.5lines)+0cm
grid.rect(height=leg$heights) 
grid.rect(height=leg$heights - unit(1,"line"), gp=gpar(lty=2))

so, indeed, it's the legend that's adding some margins (0.5 + 0.5 = 1 line in total). I reckon it's a missing guide.margin option in the theme, that is being replaced by a default value of half a line.