Why Interpolation makes ListPlot3D slow?

Let me give the idea for another approach which (at least on my machine) does not lag. The key problem seems to be when you try to rotate a complex graphic. Therefore, the basic idea behind the following is to use a simple graphic for the rotation and transfer the values to your complex graphic. The surprising thing is, that when you ControlActive inside this dynamic environment, the change to the simple form of your plot happens instantly.

In the following, please use the cuboid for rotation and see what happens:

With[{ls = Table[Sinc[x*y], {x, -10, 10, 0.2}, {y, -10, 10, 0.2}]},
 With[{
   speed = ListPlot3D[ls, PlotRange -> All, PlotStyle -> None],
   quality = 
    ListPlot3D[ls, PlotRange -> All, InterpolationOrder -> 2]
   },
  DynamicModule[{vp = {1.3, -2.4, 2}, vc = {0.5, 0.5, 0.5}, 
    vv = {0, 0, 1}},
   Column[
    {
     Graphics3D[Cuboid[{0, 0, 0}], Boxed -> False, 
      SphericalRegion -> True, 
      Epilog -> Circle[Scaled[{0.5, 0.5}], 0.5], 
      ViewPoint -> Dynamic[vp], ViewCenter -> Dynamic[vc], 
      ViewVertical -> Dynamic[vv]],
     Dynamic[
      Deploy@Show[ControlActive[speed, quality], ViewPoint -> vp, 
        ViewCenter -> vc, ViewVertical -> vv, SphericalRegion -> True]]
     }]
   ]
  ]
 ]

You could switch to a faster version of the plot when rotating with something like this:

qualityplot = ListPlot3D[ls, PlotRange -> All, InterpolationOrder -> 2];
speedplot = ListPlot3D[ls, PlotRange -> All];

DynamicModule[{displayed = qualityplot},
 EventHandler[Dynamic[displayed], {
   "MouseDown" :> (displayed = speedplot),
   "MouseUp" :> (displayed = qualityplot)},
  PassEventsDown -> True]]

It's very laggy switching from one state to the other though, so you might find it just as annoying as before.