How can I make atomic incremental backups of a running linux system using freeware?

As you mentioned, BTRFS can do this. This is how I regularly backup my laptop (which has an uptime of 9 weeks, 5 days as I type this).

Within my BTRFS filesystem, I have subvolumes. The way that you split your data into subvolumes and how you nest them is unimportant here, so long as you aren't using the root of the filesystem to store data that you want to back up.

The following commands are to illustrate syntax and possibilities, I recommend wrapping them up in a script that runs as a cronjob or systemd.timer.

To snapshot a subvolume:

btrfs subvolume snapshot -r <source> <dest>

To serialise a snapshot:

btrfs send <snapshot>

To serialise a snapshot relative to an older one (i.e. differential):

btrfs send -p <start> <end>

To generate a diff, compressing on the fly, and sending to backup server, with "progress" monitoring:

btrfs send -p <start> <end> | \
    pv -bart | \
    pbzip2 --best | \
    ssh [email protected] "cat > /backups/name.bz2"

To do similar, but re-create the BTRFS subvolumes on the backup server rather than just compressed BTRFS streams:

btrfs send -p <start> <end> | \
    pv -bart | \
    pbzip2 --best | \
    ssh [email protected] "pbzip2 -d | \
    btrfs receive <target>"

To restore, apply your snapshots in order to a new BTRFS filesystem, via btrfs receive.

---

Here is more information about BTRFS Incremental Backups

As @mark-k-cowan said you can do this with btrfs snapshots. btrfs is too immature for me. Using zfs and its snapshotting would be better. As @brady-dean said LVM snapshots are a filesystem-agnostic method. LVM snapshots can have a huge negative performance impact.

Acronis has a kernel module that works at the block layer to create snapshots. The Datto Block Driver is a GPLv2 (ahem "freeware") kernel module that does the same thing. (Disclaimer: I have never used it.)