How to speed-up nested loop?
One speedup would be to remove the sum
portion, as in this implementation it sums a list of length 2 through INTERVAL_LENGTH
. Instead, just add volume[j+1]
to the previous result of vol from the last iteration of the loop. Thus, you're just adding two integers each time instead of summing an entire list AND slicing it each time. Also, instead of starting by doing sum(volume[i+1:j+1])
, just do vol = volume[i+1] + volume[j+1]
, as you know the initial case here will always be just two indices.
Another speedup would be to use .extend
instead of .append
, as the python implementation has extend
running significantly faster.
You could also break up the final if
statement so as to only do certain computation if required. For instance, you know if vol <= 100
, you don't need to compute ret
.
This doesn't answer your problem exactly, but I think especially with the sum issue that you should see significant speedups with these changes.
Edit - you also don't need len
, since you know specifically the length of the list already (unless that was just for the example). Defining it as a number rather than len(something)
is always faster.
Edit - implementation (this is untested):
ARRAY_LENGTH = 500000
INTERVAL_LENGTH = 15
close = np.array( xrange(ARRAY_LENGTH) )
volume = np.array( xrange(ARRAY_LENGTH) )
close, volume = close.astype('float64'), volume.astype('float64')
results = []
ex = results.extend
for i in xrange(ARRAY_LENGTH - INTERVAL_LENGTH):
vol = volume[i+1]
for j in xrange(i+1, i+INTERVAL_LENGTH):
vol += volume[j+1]
if vol > 100:
ret = close[j] / close[i]
if 1.0001 < ret < 1.5:
ex( [i, j, ret, vol] )
print results
Update: (almost) completely vectorized version below in "new_function2"...
I'll add comments to explain things in a bit.
It gives a ~50x speedup, and a larger speedup is possible if you're okay with the output being numpy arrays instead of lists. As is:
In [86]: %timeit new_function2(close, volume, INTERVAL_LENGTH)
1 loops, best of 3: 1.15 s per loop
You can replace your inner loop with a call to np.cumsum()... See my "new_function" function below. This gives a considerable speedup...
In [61]: %timeit new_function(close, volume, INTERVAL_LENGTH)
1 loops, best of 3: 15.7 s per loop
vs
In [62]: %timeit old_function(close, volume, INTERVAL_LENGTH)
1 loops, best of 3: 53.1 s per loop
It should be possible to vectorize the entire thing and avoid for loops entirely, though... Give me an minute, and I'll see what I can do...
import numpy as np
ARRAY_LENGTH = 500000
INTERVAL_LENGTH = 15
close = np.arange(ARRAY_LENGTH, dtype=np.float)
volume = np.arange(ARRAY_LENGTH, dtype=np.float)
def old_function(close, volume, INTERVAL_LENGTH):
results = []
for i in xrange(len(close) - INTERVAL_LENGTH):
for j in xrange(i+1, i+INTERVAL_LENGTH):
ret = close[j] / close[i]
vol = sum( volume[i+1:j+1] )
if (ret > 1.0001) and (ret < 1.5) and (vol > 100):
results.append( (i, j, ret, vol) )
return results
def new_function(close, volume, INTERVAL_LENGTH):
results = []
for i in xrange(close.size - INTERVAL_LENGTH):
vol = volume[i+1:i+INTERVAL_LENGTH].cumsum()
ret = close[i+1:i+INTERVAL_LENGTH] / close[i]
filter = (ret > 1.0001) & (ret < 1.5) & (vol > 100)
j = np.arange(i+1, i+INTERVAL_LENGTH)[filter]
tmp_results = zip(j.size * [i], j, ret[filter], vol[filter])
results.extend(tmp_results)
return results
def new_function2(close, volume, INTERVAL_LENGTH):
vol, ret = [], []
I, J = [], []
for k in xrange(1, INTERVAL_LENGTH):
start = k
end = volume.size - INTERVAL_LENGTH + k
vol.append(volume[start:end])
ret.append(close[start:end])
J.append(np.arange(start, end))
I.append(np.arange(volume.size - INTERVAL_LENGTH))
vol = np.vstack(vol)
ret = np.vstack(ret)
J = np.vstack(J)
I = np.vstack(I)
vol = vol.cumsum(axis=0)
ret = ret / close[:-INTERVAL_LENGTH]
filter = (ret > 1.0001) & (ret < 1.5) & (vol > 100)
vol = vol[filter]
ret = ret[filter]
I = I[filter]
J = J[filter]
output = zip(I.flat,J.flat,ret.flat,vol.flat)
return output
results = old_function(close, volume, INTERVAL_LENGTH)
results2 = new_function(close, volume, INTERVAL_LENGTH)
results3 = new_function(close, volume, INTERVAL_LENGTH)
# Using sets to compare, as the output
# is in a different order than the original function
print set(results) == set(results2)
print set(results) == set(results3)