Merge two BibTeX files

bibtool -s bibliography1.bib bibliography2.bib will merge two bib files, keeping duplicate entries. bibtool -s -d bibliography1.bib bibliography2.bib will merge two bib files, commenting out one of the duplicated entries (not sure which one). For more info, see documentation.

Bibtool is on CTAN. Not to be confused with bibtools which is also on CTAN and probably also has the capacity to do this sort of thing...

Bibtool can also be found in Ubuntu repositories. (I wasn't able to compile the one from CTAN)

JabRef is a bibliography manager that has tools for merging bib-files. Or rather, for importing one file into another: File --> Import into current database. Not sure if it checks for duplicates on merge, but there is a tool for checking for duplicates as well: Tools --> Scan database --> Find duplicates.

I think there is no easy way to do this. Though it is possible to write a perl script alike to use regex to do this. But the problem is

1. Sometimes two entries are duplicates though they are not exactly the same, for example different capitalization in titles or extra {and }. So we have to workaround this by define some threshold for difference.

2. Sometimes two entries from the same author of the same year with a little difference in the title are indeed two papers. This make the threshold in (1) hard to define.

So I suggest you can only do this by hand. A relative easier way is do it with a BibTeX editor, for example BibDesk on Mac OS X. Import both files into it, and it shall warn you when two entries have the same citekey. So for those entries with same citekeys, which are almost surely duplicates, you can clean them up easily. If I remember correctly BibDesk and other editors do have this functionality. After this, sort all entries by title, and spot further duplicates by your eyes.