How to use the past with HuggingFace Transformers GPT-2?

I believe the problem is that context contains integer values exceeding vocabulary size. My assumption is based on the last traceback line:

return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self