How to create a lagged data structure using pandas dataframe

As mentioned, it could be worth looking into the rolling_ functions, which will mean you won't have as many copies around.

One solution is to concat shifted Series together to make a DataFrame:

In [11]: pd.concat([s, s.shift(), s.shift(2)], axis=1)
Out[11]: 
   0   1   2
1  5 NaN NaN
2  4   5 NaN
3  3   4   5
4  2   3   4
5  1   2   3

In [12]: pd.concat([s, s.shift(), s.shift(2)], axis=1).dropna()
Out[12]: 
   0  1  2
3  3  4  5
4  2  3  4
5  1  2  3

Doing work on this will be more efficient that on lists...

Very simple solution using pandas DataFrame:

number_lags = 3
df = pd.DataFrame(data={'vals':[5,4,3,2,1]})
for lag in xrange(1, number_lags + 1):
    df['lag_' + str(lag)] = df.vals.shift(lag)

#if you want numpy arrays with no null values: 
df.dropna().values for numpy arrays

for Python 3.x (change xrange to range)

number_lags = 3
df = pd.DataFrame(data={'vals':[5,4,3,2,1]})
for lag in range(1, number_lags + 1):
    df['lag_' + str(lag)] = df.vals.shift(lag)

print(df)

   vals  lag_1  lag_2  lag_3
0     5    NaN    NaN    NaN
1     4    5.0    NaN    NaN
2     3    4.0    5.0    NaN
3     2    3.0    4.0    5.0
4     1    2.0    3.0    4.0

For a dataframe df with the lag to be applied on 'col name', you can use the shift function.

df['lag1']=df['col name'].shift(1)
df['lag2']=df['col name'].shift(2)