Poor IO due to LUKS/Software RAID/LVM ordering?

Your layering is suboptimal because putting the raid 5 on top of the encryption means that you increase the number of encrypt/decrypt operations by 25 % - since 4 * 4 TB are encrypted.

When putting the encryption on top of the raid 5 only 3 * 4 TB are encrypted.

The reasoning behind that is: you don't have to encrypt parity data (which takes up 4 TB in your example) of encrypted data because it does not increase your security.

Your presumption about multiple kcrypt processes is just that. When basing decisions on it, it is a premature optimization that may have quite the opposite effect. Your i7 is quite beefy, probably even including some special instructions that help to speed up AES - and the Linux kernel includes several optimized variants of cryptographic primitives that are automatically selected during boot.

You can verify if the optimized routines for your CPU are used via looking at /proc/cpuinfo (e.g. flag aes there), /proc/crypto, lsmod (unless the aes modules are compiled into the kernel) and the kernel log.

You should benchmark the throughput of kryptd without involving any slow disks to see what the upper bound really is (i.e. on a RAM disk using iozone).

To be able to diagnose potential performance issues later it is also useful to benchmark your RAID-setup of choice without any encryption to get an upper bound on that end.

In addition to the crypto topic, RAID 5 involves more IO-Operations than RAID 1 or 10. Since storage is kind of cheap perhaps it is an option to buy more harddisks and use another RAID level.

I would Raid 1+0 [a2,b2]+[c2,d2], then LVM over LUKS.

Example

$ sudo mdadm --create /dev/md0 -v --raid-devices=4 \
      --level=raid10 /dev/sdb1 /dev/sdc1 /dev/sde1 /dev/sde1

NOTE: Structuring it this way will create a stripe of mirrors allowing a maxium of 2 disks to fail (one in each mirror max) and it will give you a total/2 space as opposed to raid5 which is total*~0.75.

Also I believe this schema is significantly faster because RAID5 is known to hurt performance, but you'll have less space available.

You can also check the cipher, although I think aes-cbc-essiv is the default and reasonably fast, but you could use aes-xts-plain which should be faster.