Speed up python's struct.unpack
Numpy lets you do this very quickly. In this case I think the easiest way is to use the ndarray
constructor directly:
import numpy as np
def with_numpy(buffer):
# Construct ndarray with: shape, dtype, buffer, offset, strides.
rotational = np.ndarray((firingBlocks,), '<H', buffer, 42+2, (100,))
distance = np.ndarray((firingBlocks,lasers), '<H', buffer, 42+4, (100,3))
intensity = np.ndarray((firingBlocks,lasers), '<B', buffer, 42+6, (100,3))
return rotational, distance*0.002, intensity
This returns separate arrays instead of the nested list, which should be much easier to process further. As input it takes a buffer
object (in Python 2) or anything that exposes the buffer interface. Unfortunately, it depends on your Python version (2/3) what objects you can use exactly. But this method is very fast:
import numpy as np
firingBlocks = 10**4
lasers = 32
packet_raw = np.random.bytes(42 + firingBlocks*100)
%timeit readDataPacket(memoryview(packet_raw))
# 1 loop, best of 3: 807 ms per loop
%timeit with_numpy(packet_raw)
# 100 loops, best of 3: 10.8 ms per loop
Compile a Struct
ahead of time, to avoid the Python level wrapping code using the module level methods. Do it outside the loops, so the construction cost is not paid repeatedly.
unpack_ushort = struct.Struct('<H').unpack
unpack_ushort_byte = struct.Struct('<HB').unpack
The Struct
methods themselves are implemented in C in CPython (and the module level methods are eventually delegating to the same work after parsing the format string), so building the Struct
once and storing bound methods saves a non-trivial amount of work, particularly when unpacking a small number of values.
You can also save some work by unpacking multiple values together, rather than one at a time:
distanceInformation, intensity = unpack_ushort_byte(firingData[startingByte:startingByte + 3])
distanceInformation *= 0.002
As Dan notes, you could further improve this with iter_unpack
, which would further reduce the amount of byte code execution and small slice operations.