How to loop through a python list in batch?
You are close.
chunks = (total_count - 1) // 50 + 1
for i in range(chunks):
batch = data[i*50:(i+1)*50]
l = [1,2,3,4,5,6,7,8,9,10]
batch_size = 3
for i in range(0, len(l), batch_size):
print(l[i:i+batch_size])
# more logic here
>>> [1,2,3]
>>> [4,5,6]
>>> [7,8,9]
>>> [10}
I think this is the most straight-forward, readable approach. If you need to retry a certain batch, you can retry inside the loop (serial) or you can open a thread per batch - depends on the application...
def chunk_list(datas, chunksize):
"""Split list into the chucks
Params:
datas (list): data that want to split into the chunk
chunksize (int) : how much maximum data in each chunks
Returns:
chunks (obj): the chunk of list
"""
for i in range(0, len(datas), chunksize):
yield datas[i:i + chunksize]
ref: https://www.codegrepper.com/code-examples/python/python+function+to+split+lists+into+batches