Is it possible to detach and reattach a ZFS disk without requiring a full resilver?

Don't go down the road of breaking the ZFS array to "rotate" disks offsite. As you've seen, the rebuild time is high and the resilvering process will read/verify the used size of the dataset.

If you have the ability, snapshots and sending data to a remote system is a clean, non-intrusive approach. I suppose you could go through the process of having a dedicated single-disk pool, copy to it, and zpool export/import... but it's not very elegant.

After further experimentation I've found a fair solution, however it comes with a significant trade-off. Disks which have been offline'd but not detached can later be brought back online with only an incremental resilvering operation ("When a device is brought online, any data that has been written to the pool is resynchronized with the newly available device."). In my tests this brings resilvering time for a 3-disk mirror down from 28 hours to a little over 30 minutes, with about 40GB of data-delta.

The trade-off is that any pool with an offline disk will be flagged as degraded. Provided there are still at least two online disks (in a mirrored pool) this is effectively a warning--integrity and redundancy remain intact.

As others mentioned this overall approach is far from ideal--sending snapshots to a remote pool would be far more suitable, but in my case is not feasible.

To summarize, if you need to remove a disk from a pool and later add it back without requiring a full resilvering then the approach I'd recommend is:

• offline the disk in the pool: zpool offline pool disk
• spin down the drive (if it is to be physically pulled): hdparm -Y /dev/thedisk
• leave the pool in a degraded state with the drive offlined
• to add the disk back to the pool: zpool online pool disk

And, since this is as-yet untested, there is the risk that the delta resilvering operation is not accurate. The "live" pool and/or the offline disks may experience issues. I'll update if that happens to me, but for now will experiment with this approach.

Update on 2015 Oct 15: Today I discovered the zpool split command, which splits a new pool (with a new name) off of an existing pool. split is much cleaner than offline and detach, as both pools can then exist (and be scrubbed separately) on the same system. The new pool can also be cleanly (and properly) export[ed] prior to being unplugged from the system.

(My original post follows below.)

Warning! Various comments on this page imply that it is (or might be) possible to zpool detach a drive, and then somehow reattach the drive and access the data it contains.

However, according to this thread (and my own experimentation) zpool detach removes the "pool information" from the detached drive. In other words, a detach is like a quick reformatting of the drive. After a detach lots of data may still be on the drive, but it will be practically impossible to remount the drive and view the data as a usable filesystem.

Consequently, it appears to me that detach is more destructive than destroy, as I believe zpool import can recover destroyed pools!

A detach is not a umount, nor a zpool export, nor a zpool offline.

In my experimentation, if I first zpool offline a device and then zpool detach the same device, the rest of the pool forgets the device ever existed. However, because the device itself was offline[d] before it was detach[ed], the device itself is never notified of the detach. Therefore, the device itself still has its pool information, and can be moved to another system and then import[ed] (in a degraded state).

For added protection against detach you can even physically unplug the device after the offline command, yet prior to issuing the detach command.

I hope to use this offline, then detach, then import process to back up my pool. Like the original poster, I plan on using four drives, two in a constant mirror, and two for monthly, rotating, off-site (and off-line) backups. I will verify each backup by importing and scrubbing it on a separate system, prior to transporting it off-site. Unlike the original poster, I do not mind rewriting the entire backup drive every month. In fact, I prefer complete rewrites so as to have fresh bits.