Graphics elements do not line up perfectly in exported PDFs

A simple workaround is to use Export instead of right clicking and choosing Save As ... The result will look like this:

Much better.

It's not clear to me why this difference exists between saving from the front end or using Export, but there is a significant difference in the quality of the output. Notice that the line widths are different too (the Export version is correct).

Conclusion: better not use the interactive Save As feature for graphics, or copy the graphic to the clipboard on OS X (which will have the same rendering imperfections).

Proof of the Szabolcs's idea:

Graphics[{PointSize[0.001], Point@RandomReal[1, {1000, 2}]}, ImageSize -> 30]

After Save As (28x28 grid):

After Export

Another workaround is printing the graphic to a file.