When using ggplot in R, how do I remove margins surrounding the plot area?

After using your code, I see more clearly what you're looking for. This:

gg <- ggplot(data=df, aes(V1, V2, color=cl[V3]))
gg + 
  geom_point() +
  labs(x = NULL, y = NULL, title = NULL) +
  scale_x_continuous(expand = c(0, 0), limits = range(df$V1)) +
  scale_y_continuous(expand = c(0, 0), limits = range(df$V2)) +
  scale_colour_manual(values = sort(c("#00000000", rainbow(35)), decreasing = FALSE)) +
  theme(
    panel.background = element_rect(fill = "transparent", colour = NA),
    plot.background = element_rect(fill = "transparent", colour = NA),
    panel.grid = element_blank(),
    panel.border = element_blank(),
    plot.margin = unit(c(0, 0, 0, 0), "null"),
    panel.margin = unit(c(0, 0, 0, 0), "null"),
    axis.ticks = element_blank(),
    axis.text = element_blank(),
    axis.title = element_blank(),
    axis.line = element_blank(),
    legend.position = "none",
    axis.ticks.length = unit(0, "null"),
    axis.ticks.margin = unit(0, "null"),
    legend.margin = unit(0, "null")
  )

you have to remove the labels, not-expand the x & y axis and set hard limits. The nulls are also important.'

This can also be done by doing gb <- ggplotGrob(gg) and manually editing the grobs & parameters, but I think this probably gets you what you need.

An approach that selects just the plot panel from the ggplot layout. It creates the ggplot, setting elements within the plot panel to element_blank, and no expansion of the x and y scales. It then creates the ggplot grob so that the plot panel only can be selected from the layout.

Minor edit: Updating to ggplot2 2.2.0

library(ggplot2)
library(grid)

max_iter=25
cl=colours()
step=seq(-2,0.8,by=0.005)
points=array(0,dim=c(length(step)^2,3))
t=0

for(a in step) {
  for(b in step+0.6) {
    x=0;y=0;n=0;dist=0
    while(n<max_iter & dist<4)  {
      n=n+1
      newx=a+x^2-y^2
      newy=b+2*x*y
      dist=newx^2+newy^2
      x=newx;y=newy
    }

    if(dist<4)  { 
      color=24 # black
    } else {
      color=n*floor(length(cl)/max_iter)
    }
    t=t+1
    points[t,]=c(a,b,color)
  }
}

df=as.data.frame(points)    

# ggplot with elements in the plot panel set to element_blank()
# and no expansion on the scales
p = ggplot(data=df, aes(V1, V2, color=cl[V3]))+ 
  geom_point() + 
  scale_x_continuous(expand = c(0,0), limits=range(df$V1)) +
  scale_y_continuous(expand = c(0,0), limits=range(df$V2))+
  theme(panel.grid=element_blank(), 
       panel.background=element_rect(fill = "transparent",colour = NA),
       panel.border=element_blank()) +
  scale_colour_manual(values=sort(c("#00000000", rainbow(35)), decreasing=FALSE))

# Get the ggplot grob
gt = ggplotGrob(p)

# Select plot panel only
#   gt = gt[6,4]    # Using index notation; OR
gt = gtable::gtable_filter(gt, "panel")  

# Draw it
grid.newpage()
grid.draw(gt)


# Set up a print method 
class(gt) = c("Panel", class(gt))
print.Panel <- function(x) {
   grid.newpage()   
   grid.draw(x)
}

gt
ggsave('mandelbrot.png', gt)

You can also use theme_nothing() from the cowplot package:

require(cowplot)
qplot(1:10, (1:10)^2, geom='line') + theme_nothing() + 
  scale_x_continuous(expand=c(0,0)) +
  scale_y_continuous(expand=c(0,0)) +
  labs(x = NULL, y = NULL)

Unfortunately, you still need to add labs(x = NULL, y = NULL), because there is no way in ggplot2's theme machinery to remove the axes completely. And you need to set expand=c(0,0) in the scale parameters to make sure the scale doesn't extend beyond your data range.

Result:

I was able to get rid of the white border by setting negative plot margins and setting the axis titles to NULL. I've marked the edits in the code below.

p = ggplot(data=df, aes(V1, V2, color=cl[V3]))+ 
  geom_point() + 
  theme(panel.background=element_blank(), 
        panel.grid.major=element_blank(), 
        panel.grid.minor=element_blank(), 
        panel.margin = unit(c(0, 0, 0, 0), "cm"),       
        axis.ticks=element_blank(), 
        axis.text.x=element_blank(), 
        axis.text.y=element_blank(), 
        axis.title.x=element_blank(), 
        axis.title.y=element_blank(),
        plot.background = element_rect(fill = "transparent",colour = NA),
        plot.margin = unit(c(-1, -1.2, -1.2, -1.5), "cm"),  # Edited code
        legend.position = 'none') +
  labs(x=NULL, y=NULL) # New code