Python - reading 10 bit integers from a binary file

Here's a generator that does the bit operations without using text string conversions. Hopefully, it's a little more efficient. :)

To test it, I write all the numbers in range(1024) to a BytesIO stream, which behaves like a binary file.

from io import BytesIO

def tenbitread(f):
    ''' Generate 10 bit (unsigned) integers from a binary file '''
    while True:
        b = f.read(5)
        if len(b) == 0:
            break
        n = int.from_bytes(b, 'big')

        #Split n into 4 10 bit integers
        t = []
        for i in range(4):
            t.append(n & 0x3ff)
            n >>= 10
        yield from reversed(t)

# Make some test data: all the integers in range(1024),
# and save it to a byte stream
buff = BytesIO()

maxi = 1024
n = 0
for i in range(maxi):
    n = (n << 10) | i
    #Convert the 40 bit integer to 5 bytes & write them
    if i % 4 == 3:
        buff.write(n.to_bytes(5, 'big'))
        n = 0

# Rewind the stream so we can read from it
buff.seek(0)

# Read the data in 10 bit chunks
a = list(tenbitread(buff))

# Check it 
print(a == list(range(maxi)))    

output

True

---

Doing list(tenbitread(buff)) is the simplest way to turn the generator output into a list, but you can easily iterate over the values instead, eg

for v in tenbitread(buff):

or

for i, v in enumerate(tenbitread(buff)):

if you want indices as well as the data values.

---

Here's a little-endian version of the generator which gives the same results as your code.

def tenbitread(f):
    ''' Generate 10 bit (unsigned) integers from a binary file '''
    while True:
        b = f.read(5)
        if not len(b):
            break
        n = int.from_bytes(b, 'little')

        #Split n into 4 10 bit integers
        for i in range(4):
            yield n & 0x3ff
            n >>= 10

We can improve this version slightly by "un-rolling" that for loop, which lets us get rid of the final masking and shifting operations.

def tenbitread(f):
    ''' Generate 10 bit (unsigned) integers from a binary file '''
    while True:
        b = f.read(5)
        if not len(b):
            break
        n = int.from_bytes(b, 'little')

        #Split n into 4 10 bit integers
        yield n & 0x3ff
        n >>= 10
        yield n & 0x3ff
        n >>= 10
        yield n & 0x3ff
        n >>= 10
        yield n 

This should give a little more speed...