Compute md5sum for each line in a file
In addition to @Ole Tange's approach, here's an optimized solution (Python's part):
md5summer.py
script:
#!/usr/bin/python
import sys
import hashlib
for r in sys.stdin:
if r.strip():
h = hashlib.md5()
h.update(r.encode());
print r, '\t', h.hexdigest()
Points of optimization:
hashlib
- using actual library instead of deprecated onefor r in sys.stdin:
- reading from generator-like object instead of listif r.strip():
- check for empty record to avoid redundanthashlib
methods invocations
Usage:
parallel --pipepart -a my80gb-file -S server1,server2 --block 1 /path_to/md5summer.py
Sample output:
a 0cc175b9c0f1b6a831c399e269772661
b 92eb5ffee6ae2fec3ad71c777531578f
c 4a8a08f09d37b73795649038408b5f33
d 8277e0910d750195b448797616e091ad
f 8fa14cdd754f91cc6554c9e71929cce7
e e1671797c52e15f763380b45e841ec32
g b2f5ff47436671b6e533d8dc3614845d
h 2510c39011c5be704182423e3a695e91
i 865c0c0b4ab0e063e5caa3387c1a8741
j 363b122c528f54df4a0446b6bab05515
k 8ce4b16b22b58894aa86c421e8759df3
l 2db95e8e1a9267b7a1188556b2013b33
m 6f8f57715090da2632453988d9a1501b
n 7b8b965ad4bca0e41ab51de7b31363a1
p 83878c91171338902e0fe0fb97a8c47a
o d95679752134a2d9eb61dbd7b91c4bcc
q 7694f4a66316e53c8cdd9d9954bd611d
r 4b43b0aee35624cd95b910189b3dc231
s 03c7c0ace395d80182db07ae2c30f034
t e358efa489f58062f10dd7316b65649e
u 7b774effe4a349c6dd82ad4f4f21d34c
v 9e3669d19b675bd57058fd4664205d2a
w f1290186a5d0b1ceab27f4e77c0c5d68
x 9dd4e461268c8034f5c8564e155c67a6
y 415290769594460e2e485922904f345d
z fbade9e36a3f36d3d676c1b808451dd7
...
First make a single threaded program (md5er
), that can generate the correct output given the input:
#!/usr/bin/python
import sys
import hashlib
for r in sys.stdin:
print r[:-1], '\t', hashlib.md5(r[:-1]).hexdigest()
Then use GNU Parallel to split the input into chunks that can be distributed to compute servers:
parallel --pipepart -a my80gb-file -Sworker1,worker2,worker3,: --block -10 md5er
This could just be a oneliner in perl:
head 80gb | perl -MDigest::MD5=md5_hex -nlE'say"$_\t".md5_hex($_)'
a 0cc175b9c0f1b6a831c399e269772661
b 92eb5ffee6ae2fec3ad71c777531578f
c 4a8a08f09d37b73795649038408b5f33
d 8277e0910d750195b448797616e091ad
e e1671797c52e15f763380b45e841ec32
f 8fa14cdd754f91cc6554c9e71929cce7
g b2f5ff47436671b6e533d8dc3614845d
h 2510c39011c5be704182423e3a695e91
i 865c0c0b4ab0e063e5caa3387c1a8741
j 363b122c528f54df4a0446b6bab05515
If you need to store the output and want a nice progress bar while chewing this huge chunk:
sudo apt install pv #ubuntu/debian
sudo yum install pv #redhat/fedora
pv 80gb | perl -MDigest::MD5=md5_hex -nlE'say"$_\t".md5_hex($_)' | gzip -1 > 80gb-result.gz