Is there a block-level storage file system?
Assuming your question is about data deduplication, there are a few file systems which support that on Linux:
- ZFS, with online deduplication (so data is deduplicated as it is stored), but with extreme memory requirements which make the feature hard to use in practice;
- Btrfs, with “only” out-of-band deduplication, albeit with tightly-integrated processes which provide reasonably quick deduplication after data is stored;
- SquashFS, but that probably doesn’t fit your requirements because it’s read-only.
XFS is supposed to get deduplication at some point, and Btrfs is also supposed to get online deduplication.
Keep an eye on Wikipedia’s file system comparison to see when this changes.
The S3QL filesystem has block level deduplication. It's promoted as an S3 cloud storage solution but it also works very nicely on local storage.
Here's an example from part of our backups/archive server
s3qlstat /path/to/some/archives
Directory entries: 12430247
Inodes: 6343756
Data blocks: 1357349
Total data size: 12.4 TB
After de-duplication: 3.84 TB (30.92% of total)
After compression: 3.71 TB (29.84% of total, 96.52% of de-duplicated)
Database size: 1.29 GiB (uncompressed)
Cache size: 0 bytes, 0 entries
Cache size (dirty): 0 bytes, 0 entries
Queued object removals: 0
The underlying storage that this filesystem uses
df -h /var/s3ql/part-of-archive
Filesystem Size Used Avail Use% Mounted on
/dev/sde 6.0T 3.8T 2.0T 66% /var/s3ql/part-of-archive
This tells me that the underlying storage is using a little under 4TB, but that it's storing around 12TB of deduplicated data. (My archives have quite a lot of duplicated blocks in them. Unsurprisingly I hope.) The compression layer is disabled here; if I was using true S3 storage I would have left it enabled.
The SQLite database that manages the filesystem itself is just over 1GB, which is quite large, but as I'm using this for archiving rather than full-on high demand production use is fine.