tool or technique to get a diff of two different linux installations
Whenever I have a good reference system and a misbehaving one, I try to compare them with vimdiff. What I compare varies with the problem, e.g.
1) When comparing servers at the package level, I create sorted lists of packages on each server, send the results to files and diff them, e.g.
On server1:
dpkg --get-selections|sort > server1_packages
On server2:
dpkg --get-selections|sort > server2_packages
Copy both files to the same machine and diff (or vimdiff) them.
2) Make a list of running services as in example 1
sysv-rc-conf --list|sort > server1_services
sysv-rc-conf --list|sort > server2_services
...etc., and vimdiff those.
3) If you are troubleshooting inconsistent configurations with Apache for example, make copies of the config files, and vimdiff those, etc.
Most of the differences between two installations of the same distribution will be in /etc. Copy the contents of /etc from one machine to a temporary directory on the other and run
diff -ru /etc /copy/of/other/etc
If you want to compare a machine with a default installation, get a copy from a fresh default installation (perhaps in a virtual machine).
If you can control a machine from the start, make sure to install etckeeper to keep /etc under version control. Then you'll be able to see precisely what's changed.
Of course, there'll be a lot of differences. If you don't yet have an idea of where to look, this is the wrong approach to investigate a particular difference in behavior. For example, let's say the command adduser behaves differently on two machines. Then the best approach is to look at what adduser is doing; run it on both machines and compare. If it has an option to tell it to be more verbose (adduser doesn't), use it. More radically, run the program under strace, e.g.
strace -s9999 -efile adduser …
to see what files adduser accesses.
To harken back to an earlier question, your nifty tool, which sounds like a good idea to me, but I've never heard of such a beast existing, needs to check things like filesystem type and sizes.
A real life example that happened to me in 2002: two people installed SuSE 7.3 systems on two adjacent rack-mounted servers. They put a lot of effort into getting the packages exactly the same on both servers. A short while later, we had problems with some software in development. It came down to ext3 filesystem on one server, Reiserfs on the other. A plain "ls" on Reiserfs gives file names in lexical order, but not on ext3. One program failed on out-of-order filenames.