Special File that causes I/O error

There's a great set of answers to this on Stack Overflow and Server Fault already but some techniques were missing. To make life easier here's a list of VM/Linux block device/Linux filesystem/Linux userspace library I/O fault injection mechanisms:

• Use Device Mapper's error/flakey/delay/dm-dust devices to return errors/corruption from, or delay/split IO to a synthesised block device (kernel, requires kernel to have been built with device mapper support, appropriate additional device mapper modules (dm-dust is only available on kernel >=5.2) and to have device mapper userspace bits).
• Use md's faulty personality to perform periodic fault injection on a synthesised block device. See the --layout option of the mdadm man page for how to configure it (kernel and mdadm userspace bits).
• Use libfiu to perform fault injection on POSIX API calls (userspace, can be used with LD_PRELOAD).
• Use the Linux kernel's fault injector to inject an error into the underlying block device (kernel, requires kernel to have been built with FAIL_MAKE_REQUEST=y).
• Using SystemTap to do fault injection (kernel, requires a kernel to have been built with lots of stuff).
• Inject filesystem faults using CharybdeFS or PetardFS (userspace via FUSE).
• Create a synthesised block device using the Linux scsi_debug driver that performs fault injection (kernel).
• Run your system within QEMU and use QEMU to inject block device errors using the blkdebug driver (VM).
• Create a synthesised block device via the null_blk device's options to inject faults (kernel >= 4.14 but options like timeout probabilities didn't arrive until 4.17 and require the kernel to have been built with BLK_DEV_NULL_BLK_FAULT_INJECTION=y).
• Create a synthesised Network Block Device which is served to the host via NBDkit filters such as delay or error and then attach a block device to it via nbd-client (kernel + NBD userspace bits, kernel >= 4.18 built with NBD support, nbdclient >= 3.18 and nbdkit >= 1.8.1 recommended - see NBDKit demo video around the 20 minute mark).

Bonus fact: SQLite has a VFS driver for simulating errors so it can get good test coverage.

Related:

• How can I simulate a failed disk during testing?
• Simulate a faulty block device with read errors?
• Generate a read error
• Intentionally cause an I/O error in Linux?

You can use dmsetup to create a device-mapper device using either the error or flakey targets to simulate failures.

dmsetup create test --table '0 123 flakey 1 0 /dev/loop0'

Where 123 is the length of the device, in sectors and /dev/loop0 is the original device that you want to simulate errors on. For error, you don't need the subsequent arguments as it always returns an error.

You want a fault injection mechanism for I/O.

On Linux, here's a method that doesn't require any prior setup and generates an unusual error (not EIO “Input/output error” but ESRCH “No such process”):

cat /proc/1234/mem

where 1234 is the PID of a process running as the same user as the process you're testing, but not that process itself. Credits to rubasov for thinking of /proc/$pid/mem.

If you use the PID of the process itself, you get EIO, but only if you're reading from an area that isn't mapped in the process's memory. The first page is never mapped, so it's ok if you read the file sequentially, but not suitable for a database process that seeks directly to the middle of the file.

With some more setup as root, you can leverage the device mapper to create files with valid sectors and bad sectors.

Another approach would be to implement a small FUSE filesystem. EIO is the default error code when your userspace filesystem driver does something wrong, so it's easy to achieve. Both the Perl and Python bindings come with examples to get started, you can quickly write a filesystem that mostly mirrors existing files but injects an EIO in carefully chosen places. There's an existing such filesystem: petardfs (article), I don't know how well it works out of the box.

Yet another method is an LD_PRELOAD wrapper. An existing one is Libfiu (fault injection in userspace). It works by preloading a library that overloads the POSIX API calls. You can write simple directives or arbitrary C code to override the normal behavior.