Panning by moving Origin (0,0) with Mouse

An illustrative example that demonstrates how the changing coordinate system can be handled.

Manipulate[
 Graphics[{Point[p], Locator[{0, 0}, Appearance -> Large], Red, AbsolutePointSize[5], 
   Point[shift]}, Axes -> True, AxesOrigin -> {0, 0}, 
  PlotRange -> {{-5, 5}, {-5, 5}} - shift - p, 
  GridLines -> {Range[-5, 5, 1], Range[-5, 5, 1]}], {{p, {0, 0}}, 
  Locator, TrackingFunction -> {None, p = #; &, (shift = shift + #; p = {0, 0}); &}},
 {{shift, {0, 0}}, None}]

The coordinate system of MousePosition is static.

Manipulate[
 Graphics[{AbsolutePointSize[5], Point[p], 
   Locator[MousePosition["Graphics", {0, 0}], Appearance -> Large, 
    Enabled -> $ControlActiveSetting], Red, Point[shift]}, 
  Axes -> True, AxesOrigin -> {0, 0}, 
  PlotRange -> {{-5, 5}, {-5, 5}} - shift - p, 
  GridLines -> {Range[-5, 5, 1], Range[-5, 5, 1]}], {{p, {0, 0}}, 
  Locator, TrackingFunction -> {None, 
    p = #; &, (shift = shift + #; p = {0, 0}); &}, 
  Appearance -> None}, {{shift, {0, 0}}, None}]

Using the static MousePosition coordinate system to drag the axis origin.

Manipulate[
 Graphics[{Point[p]}, Axes -> True, AxesOrigin -> {0, 0}, PlotRange -> pr, 
  GridLines -> {Range[-5, 5, 1], Range[-5, 5, 1]}], 
 {{p, {0, 0}}, Locator, 
  TrackingFunction -> (pr = pr - MousePosition["Graphics", {0, 0}]; &)}, 
 {{pr, {{-5, 5}, {-5, 5}}}, None}]

Getting rid of the extra Manipulate variable.

Manipulate[
 Graphics[{Point[p], Locator[{0, 0}]}, Axes -> True, 
  AxesOrigin -> {0, 0}, PlotRange -> {{-5, 5}, {-5, 5}} - p, 
  GridLines -> {Range[-5, 5, 1], Range[-5, 5, 1]}, 
  ImageSize -> Medium], 
 {{p, {0, 0}}, Locator, TrackingFunction -> (p = p + MousePosition["Graphics", {0, 0}]; &), 
  Appearance -> None}]

An alteration using scaled coordinates, a changing MouseAppearance at the position of the Locator, and a limitation of the dragging area to the area of the Graphics object.

Manipulate[
 Graphics[{MouseAppearance[Locator[Scaled[p]], "DragGraphics"], 
   Transparent, AbsolutePointSize[7], MouseAppearance[Point[Scaled[p]], "DragGraphics"]}, 
  Axes -> True, AxesOrigin -> {0, 0}, 
  PlotRange -> {{-5, 5}, {-5, 5}} - 10*(p - 0.5), 
  GridLines -> {Range[-5, 5, 1], Range[-5, 5, 1]}, 
  ImageSize -> Medium], 
 {{p, {0.50, 0.50}}, Locator, 
  TrackingFunction -> (If[MousePosition["GraphicsScaled", {0, 0}] ∈ Rectangle[],
        p = MousePosition["GraphicsScaled", {0, 0}]]; &), 
  Appearance -> None}]

I think this is easier to do with a dynamic module than with a manipulate expression. Here is my implementation using DynamicModule. Note that the locator is constrained to snap to the nearest grid point.

With[{span = 10.},
  DynamicModule[{origin, xmin, xmax, ymin, ymax},
    origin = {0, 0};
    {xmin, xmax} = {ymin, ymax} = span {-1., 1.}/2.;
    Dynamic @
      Graphics[
         Locator[
           Dynamic[
             origin, 
             {Automatic, 
              Module[{x, y},
                {x, y} = #;
                xmin -= Round[x]; xmax -= Round[x]; 
                ymin -= Round[y]; ymax -= Round[y]; 
                origin = {0, 0}] &}]],
         Axes -> True,
         AxesOrigin -> {0, 0},
         PlotRange -> {{xmin, xmax}, {ymin, ymax}},
         GridLines -> {Range[xmin, xmax], Range[ymin, ymax]}]]]

Here is the initial view

and the view after the locator has been moved to {2, 1} and released.