dd on entire disk, but do not want empty portion
Solution 1:
Back in the day I ran into a similar problem with embedded Linux distributions - get rid of all the junk before compressing the image.
dd if=/dev/zero of=asdf.txt
. Wait until it dies. Delete asdf.txt.
You've just written zeros to all free space on the device.
Now take a disk image and run it through gzip. Voila, sparse image.
Probably doesn't scale very well and could cause problems if you actually need to write to the disk, but hey.
You could take an rsync snapshot of the disk to another volume, zero that, and then take that disk image.
Note: Could be hazardous for SSD, user should consider this operation befor committing.
Solution 2:
Assuming you want to save /dev/sdXN
to /tgtfs/image.raw
and you are root:
mkdir /srcfs && mount /dev/sdXN /srcfs
Use
zerofill
or just:
dd if=/dev/zero of=/srcfs/tmpzero.txt
to fill unused blocks with zero; wait for it to fill the file system completely then:
rm /srcfs/tmpzero.txt
Take the image with dd and use conv=sparse to punch zeros on-the-fly:
dd conv=sparse if=/dev/sdxn of=/tgtfs/image.raw
If you want to use compression you don't need to punch the zeros with dd as zero blocks are highly compressible:
dd if=/dev/sdxn | gz -c | dd of=/tgtfs/image.raw
PS: You should note that it is not a good idea to this (filling the file system with zeros) on a flash memory based storage media (i.e. your source file system being a SSD) on a regular basis, as it will cause extensive write to your SSD and reduce its lifespan. (but it's alright for occasional transfer of data)
Solution 3:
Use dd, with the count option.
In your case you were using fdisk so I will take that approach. Your "sudo fdisk -l "produced:
Disk /dev/sda: 64.0 GB, 64023257088 bytes
255 heads, 63 sectors/track, 7783 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0000e4b5
Device Boot Start End Blocks Id System
/dev/sda1 * 1 27 209920 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 27 525 4000768 5 Extended
Partition 2 does not end on cylinder boundary.
/dev/sda5 27 353 2621440 83 Linux
/dev/sda6 353 405 416768 83 Linux
/dev/sda7 405 490 675840 83 Linux
/dev/sda8 490 525 282624 83 Linux
The two things you should take note of are 1) the unit size, and 2) the "End" column. In your case you have cylinders that are equal to 8225280 Bytes. In the "End" column sda8 terminates at 525 (which is 525[units]*16065*512 = ~4.3GB)
dd can do a lot of things, such as starting after an offset, or stopping after a specific number of blocks. We will do the latter using the count option in dd. The command would appear as follows:
sudo dd if=/dev/sda of=/your_directory/image_name.iso bs=8225280 count=526
Where -bs is the block size (it is easiest to use the unit that fdisk uses, but any unit will do so long as the count option is declared in these units), and count is the number of units we want to copy (note that we increment the count by 1 to capture the last block).
Solution 4:
While /dev/zero
ing the free-disk-space and use dd conv=sparse
/gz -c
is possible, on huge disks with empty space running in 100s of GBs, /dev/zero
ing is painfully slow - not to mention that as other answers noted, /dev/zero
ing an SDD till EOF.
Here's what I did when I ran into this situation:
On a lubuntu live CD, used
gparted
to 'shrink' the disk to minimum possible size, leaving rest of the space unallocatedUsed
dd bs=1M count=<size_in_MBs> if=/dev/sdX | gzip -c --fast| dd of=/path/to/image.gz
to create the fast-compressed image (needless to say, you may want to skip the compression if you have sufficient space to store raw data (or are otherwise inclined to reduce CPU loading)- Used
dd if=/path/to/image.gz | gunzip -c | dd bs=1M of=/dev/sdY
to copy the data back onto different disk - Used
gparted
again to 'expand' the partition
I haven't tried it for multiple partitions but I believe the process above can be adapted to copy 'partitions' if partition-table on destination disk is created first and only the data contained in the partition is copied via dd
- reading/writing offsets (skip
/seek
option of dd
, respectively) would be required as appropriate.
Solution 5:
You can't. dd
is a very low level tool and it has no means of distinguishing between files and empty space.
On the other hand the empty space will compress very, very nicely so if you are only concerned about storage space, not for example write time, then just pipe it through gzip.