What's the difference between creating mdadm array using partitions or the whole disks directly
The most important difference is that it allows you to increase the flexibility for disk replacement. It is better detailed below along with a number of other recommendations.
One should consider to use a partition instead of the entire disk. This should be under the general recommendations for setting up an array and may certainly spare you some headaches in the future when further disk replacements get necessary.
The most important arguments is:
Disks from different manufacturers (or even different models of the "same" capacity from the same manufacturer) don't necessarily have the exact same disk size and, even the smallest size difference, will prevent you from replacing a failed disk with a newer one if the second is smaller than the first. Partitioning allows you to workaround this;
Side note on why to use different manufacturers disks: Disks will fail, this is not a matter of a "if" but a "when". Disks of the same manufacturer and the same model have similar properties, and so, higher chances of failing together under the same conditions and time of use. The suggestion so is to use disks from different manufacturers, different models and, in special, that do not belong to the same batch (consider buying from different stores if you are buying disks of the same manufacturer and model). This is not uncommon that a second disk fail happen during a resotre after a disk replacement when disks of the same batch are used. You certainly don't want this to happen to you.
So the recommendations:
1) Partition the disks that will be used with a slightly smaller capacity than the overall disk space (e.g, I have a RAID5 array of 2TB disks and I intentionally partitioned them wasting about 100MB in each). Then, use /dev/sd?1 of each one for composing the array - This will add a safety margin in case a new replacing disk has less space than the original ones used to assemble the array when it was created;
2) Use disks from different manufacturers;
3) Use disks of different models if different manufacturers are not an option for you;
4) Use disks from different batches;
5) Proactively replace disks before they fail and not all at the same time. This may be a little paranoid and really depends on the criticity of the data you have. I use to have disks that have 6 months differences in age from each other;
6) Make regular backups (always, regardless if you use an array or not). Raid doesn't serve the same purpose of backups. Arrays assure you high availability, Backups allow you to restore lost files (including the ones that get accidentally deleted or are damaged by viruses, some examples of something that using arrays will not protect you from).
OBS: Except for all the non-neglectable rational above, there aren't much further technical differences between using /dev/sd? vs /dev/sd?#.
Good luck
Another important argument is that some mainboards may delete your RAID superblocks if you use whole disk devices and are not super careful with wiping them when adding disks to a RAID array that once were GPT devices.
I learned this the hard way, when my ASRock mainboard's UEFI rendered my RAID unusable:
- http://forum.asrock.com/forum_posts.asp?TID=10174
- https://news.ycombinator.com/item?id=18541493
- https://www.phoronix.com/scan.php?page=news_item&px=Linux-Software-RAID-ASRock
To protect yourself against this, always run sgdisk --zap on a formerly GPT-formatted disk that you want to use for a whole-device RAID.
wipefs -a may also work to wipe all kinds of leftover partitioning info (not only GPT) before adding the device to the disk.
That said, just using partitions seems like a best practice to make this less likely to happen.
I never knew what the real arguments were for this best practice (beyond @Marcelo's answer explaining that it can deal with disks of slightly different sizes). Now I know.
By the way, if this happens to you, your data is not lost. You can most likely just sgdisk --zap the device, and then recreate the RAID with e.g. mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdc /dev/sdd (mdadm will tell you that it already detects past data, and asks you if you want to continue re-using that data). I tried this mutliple times and it worked, but I still recommend taking a backup before you do it.
There are couple of posts citing problem with using the whole device as a RAID member instead of partition, one here. I can confirm that to be the case.
So, if you create RAID using the whole device (say, /dev/sda, /dev/sdb, ... instead of on /dev/sda1, /dev/sdb1, ...) the RAID device doesn't reassemble after a reboot even if you have configuration saved under mdadm.conf. There is also a possibility of mainboard firmware overwriting or removing the RAID superblock. In my setup, running mdadm --assemble --scan --verbose doesn't get me back my RAID setup and it seems to be the case of RAID superblock not being available.