Creating a pandas pivot table to count number of times items appear in a list together

Here is another way by using numpy broadcasting to create a matrix which is obtained by comparing each value in user_id with every other value, then create a new dataframe from this matrix with index and columns set to page_view_page_slug and take sum on level=0 along axis=0 and axis=1 to count the user_ids of the cross section of slugs:

a = df['user_id'].values
i = list(df['page_view_page_slug'])

pd.DataFrame(a[:, None] == a, index=i, columns=i)\
   .sum(level=0).sum(level=0, axis=1).astype(int)

---

       slug1  slug2  slug3  slug4  slug5
slug1      2      2      2      1      1
slug2      2      2      2      1      1
slug3      2      2      2      1      1
slug4      1      1      1      1      0
slug5      1      1      1      0      1

Let's try groupby and reduce:

from functools import reduce

dfs = [pd.DataFrame(1, index=list(s), columns=list(s)) 
      for _, s in df.groupby('user_id')['page_view_page_slug']]
      
df_out = reduce(lambda x, y: x.add(y, fill_value=0), dfs).fillna(0).astype(int)

Details:

group the dataframe on user_id then for each group in page_view_page_slug per user_id create an adjacency dataframe with index and columns corresponding to the slugs in that group.

>>> dfs

[       slug1  slug2  slug3  slug4
 slug1      1      1      1      1
 slug2      1      1      1      1
 slug3      1      1      1      1
 slug4      1      1      1      1,
        slug5  slug3  slug2  slug1
 slug5      1      1      1      1
 slug3      1      1      1      1
 slug2      1      1      1      1
 slug1      1      1      1      1]

Now reduce the above adjacency dataframes using a reduction function DataFrame.add with optional parameter fill_value=0 so as to count the user_ids of the cross section of slugs.

>>> df_out

       slug1  slug2  slug3  slug4  slug5
slug1      2      2      2      1      1
slug2      2      2      2      1      1
slug3      2      2      2      1      1
slug4      1      1      1      1      0
slug5      1      1      1      0      1

---

Optionally you can wrap the above code in a function as follows:

def count():
    df_out = pd.DataFrame()
    for _, s in df.groupby('user_id')['page_view_page_slug']:
        df_out = df_out.add(
            pd.DataFrame(1, index=list(s), columns=list(s)), fill_value=0)

    return df_out.fillna(0).astype(int)

>>> count()

       slug1  slug2  slug3  slug4  slug5
slug1      2      2      2      1      1
slug2      2      2      2      1      1
slug3      2      2      2      1      1
slug4      1      1      1      1      0
slug5      1      1      1      0      1