O_DIRECT vs. O_SYNC on Linux/FreeBSD

With current harddisks, there is no assurance that a file is actually written to disk even if the disk reports the write as complete to the OS! This is due to built-in cache in the drive.

On freeBSD you can disable this by setting the kern.cam.ada.write_cache sysctl to 0. This will degrade write performance significantly. Last time I measured it (WDC WD5001ABYS-01YNA0 harddisk on an ICH-7 chipset, FreeBSD 8.1 AMD64), continuous write performance (measured with dd if=/dev/zero of=/tmp/foo bs=10M count=1000) dropped from 75,000,000 bytes/sec to 12,900,000 bytes/sec.

If you want to be absolutely sure that your files are written;

• Disable write caching with sysctl kern.cam.ada.write_cache=0 followed by camcontrol reset <bus>:<target>:<lun>.
• Open the file with the O_SYNC option.

Note:

• Your write perfomance (on a HDD) will now absolutely suck.
• Do not mount the partition with the sync option; that will cause all I/O (including reads) to be done syncronously.
• Do not use O_DIRECT. It will try to bypass the cache altogether. That will probably also influence reads.

O_DIRECT basically exists solely for Oracle to bypass the kernel's buffer cache layer and do its own caching. It has ill-defined semantics, arbitrary limitations on the size and alignment of reads you can perform, and generally should not be used. O_SYNC is supposed to give you the effects you want, but without an underlying filesystem that's robust against power failure or crashes, it still might not be sufficient for your needs.