Diff of two similar big raw binary files

For the second application/issue, I would use a deduplicating backup program like restic or borgbackup, rather than trying to manually keep track of "patches" or diffs. The restic backup program allows you to back up directories from multiple machines to the same backup repository, deduplicating the backup data both amongst fragments of files from an individual machine as well as between machine. (I have no user experience with borgbackup, so I can't say anything about that program.)

Calculating and storing a diff of the abc and abc2 files can be done with rsync.

This is an example with abc and abc2 being 153 MB. The file abc2 has been modified by overwriting the first 2.3 MB of the file with some other data:

$ ls -lh
total 626208
-rw-r--r--  1 kk  wheel   153M Feb  3 16:55 abc
-rw-r--r--  1 kk  wheel   153M Feb  3 17:02 abc2

We create out patch for transforming abc into abc2 and call it abc-diff:

$ rsync --only-write-batch=abc-diff abc2 abc

$ ls -lh
total 631026
-rw-r--r--  1 kk  wheel   153M Feb  3 16:55 abc
-rw-------  1 kk  wheel   2.3M Feb  3 17:03 abc-diff
-rwx------  1 kk  wheel    38B Feb  3 17:03 abc-diff.sh
-rw-r--r--  1 kk  wheel   153M Feb  3 17:02 abc2

The generated file abc-diff is the actual diff (your "patch file"), while abc-diff.sh is a short shell script that rsync creates for you:

$ cat abc-diff.sh
rsync --read-batch=abc-diff ${1:-abc}

This script modifies abc so that it becomes identical to abc2, given the file abc-diff:

$ md5sum abc abc2
be00efe0a7a7d3b793e70e466cbc53c6  abc
3decbde2d3a87f3d954ccee9d60f249b  abc2
$ sh abc-diff.sh
$ md5sum abc abc2
3decbde2d3a87f3d954ccee9d60f249b  abc
3decbde2d3a87f3d954ccee9d60f249b  abc2

The file abc-diff could now be transferred to wherever else you have abc. With the command rsync --read-batch=abc-diff abc, you would apply the patch to the file abc, transforming its contents to be the same as the abc2 file on the system where you created the diff.

Re-applying the patch a second time seems safe. There is no error messages nor does the file's contents change (the MD5 checksum does not change).

Note that unless you create an explicit "reverse patch", there is no way to easily undo the application of the patch.

---

I also tested writing the 2.3 MB modification to some other place in the abc2 data, a bit further in (at about 50 MB), as well as at the start. The generated "patch" was 4.6 MB large, suggesting that only the modified bits were stored in the patch.

How to compute a binary diff of abc and abc2?

Using bsdiff/bspatch or xdelta and others.

$ bsdiff older newer patch.bin     # patch.bin is created
[...]
$ bspatch older newer patch.bin    # newer is created

However, these admonishments from the man pages are to be noted:

• bsdiff uses memory equal to 17 times the size of oldfile, and requires an absolute minimum working set size of 8 times the size of oldfile.
• bspatch uses memory equal to the size of oldfile plus the size of newfile, but can tolerate a very small working set without a dramatic loss of performance.

Have you tried just forcing diff to treat the files as text:

diff -ua abc abc2

As explained here.

• -u output NUM (default 3) lines of unified context
• -a treat all files as text

This should get you a patch. The downside of this is the 'lines' could be quite long and that could bloat the patch.