A very, very, very big div

Changing to position: fixed seems to speed things up.

Use transform instead of top/left:

container.style.transform = 'translate(' + container.x + 'px, ' + container.y + 'px)';

A live demo at jsFiddle.