ZFS and SAN -- best practices?
Solution 1:
FWIW I have experience with up to 92 disks in a single ZFS pool and so far it works fine.
But if you're really talking about several hundreds of disks I would consider partitioning them into a small number of disjunct (but still large) pools. I don't want to know how long e.g. a zpool scrub runs on a 3000 disk pool (but you want to scrub regularly). Also the output of commands like zpool status would be unwieldy with such a large number of disks. So why put all eggs into a single basket?
(Side note on dedup: Notice that although dedup can be controlled at the dataset level it will find duplicates at the pool level. I.e. you'll probably get worse dedup results if you're partitioning as suggested. On the other hand you'll need much more memory to hold the dedup hashes of a single giant pool which might not fit into ARC+L2ARC if the pool's too big. So if you are using dedup the amount of available memory is probably a good indicator for the maximum practical pool size.)
Solution 2:
We let our SANs manage the RAID. Why spend money on all that battery backed NVRAM and those dedicated processors and then offload the work onto the server, whose CPUs I want doing something other than RAID checksums?
Solution 3:
Here is a website you may want to look at to consider size and configuration of the pool(s) for probability of data loss.
https://blogs.oracle.com/relling/entry/zfs_copies_and_data_protection
Solution 4:
It's an old question but it's just a relevant today as it was 7 years ago!
To answer the first part of the question, I'm not aware of what we would call a "SAN" that would ever expose the raw disks to a server that could run ZFS. A SAN by definition only presents block storage (LUNs) or perhaps with something like a Filer/FS presents a NFS or CIFS. There are some "SANs" that actually run ZFS internally, but this is largely abstracted away - the disks are never exposed to a server, instead the "Filer" component of the SAN presents block or network file systems to servers.
A device that presents the raw disks (over SAS or less likely over FC) is a DAS. To run ZFS, typically you'd be telling the RAID controller to present the disks as a JBOD.
However should you use ZFS on a LUN presented by a SAN? Possibly: ZFS vs for instance EXT4, provides a few extra features such as scrubs that check checksums or for running snapshots. A scrub probably can't auto heal in the same way that it can if it is doing the disk RAID, but it can still alert you to corruption, helping to prevent bitrot. The snapshots you can create on, for example, a Linux SAMBA fileserver are vastly superior to what you can do with EXT, these can even be exposed in Windows as "Previous Versions).