Pythonic way to aggregate object properties in memory efficient way?
You can use operator.attrgetter
in order to get multiple attribute of your objects then use itertools.zip_longest
(itertools.izip_longest
in Python 2.X ) to attach the relative attributes together.
from operator import attrgetter
all_result = [attrgetter('usedBytes','total','aligned','encoding')(obj) for obj in keys.items()]
Or use a generator expression to create a generator instead of a list :
all_result = (attrgetter('usedBytes','total','aligned','encoding')(obj) for obj in keys.items())
Then use zip_longest
:
used_bytes, total_bytes, aligned_bytes, encodings = zip_longest(*all_results)
Then use map
function to apply the sum
function on iterables for which you need the sum:
used_user, used_real, aligned = map(sum,(used_bytes, total_bytes, aligned_bytes))
And separately for len
and mean
:
total_elements = len(used_bytes)
mean = statistics.mean(used_bytes)
And if you want to handle all the sub lists as generator (which is more optimized in terms of memory use and less performance in terms of runtime) you can use a new class in order to calculate the desire result separately using generators :
from itertools import tee
class Aggregator:
def __init__(self, all_obj):
self.obj = all_obj
self.used_user, self.mean = self.getTotalBytesAndMean()
self.total_elements = len(self.all_obj)
self.aligned = self.getAligned()
def getTotalBytesAndMean(self):
iter_1, iter_2 = tee((obj.usedBytes for obj in self.all_obj))
return sum(iter_1), statistics.mean(iter_2)
def getTotal(self):
return sum(obj.total for obj in self.all_obj)
def getAligned(self):
return sum(obj.aligned for obj in self.all_obj)
def getEncoding(self):
return (obj.encoding for obj in self.all_obj)
Then you can do :
Agg = Aggregator(keys.items())
# And simply access to attributes
Agg.used_user