How do you glue a text to a point?

First of all your Offset specification is strange, I'd make it Offset[{-5, -5}, {2, 1}], right?

Then, you can use GraphicsGroup to treat it as a single item for the purpose of interactive editing:

p0 = Point[{0, 0}];
p1 = Point[{2, 1}];
Graphics[{
  GraphicsGroup @ {p0, Text["A", Offset[{-5, -5}, {0, 0}]]}, 
  GraphicsGroup @ {p1, Text["B", Offset[{-5, -5}, {2, 1}]]}
}, ImageSize -> {200, 100}]

This is just a little remark. Using Kuba's answer above we can define a help function

namedPoint[pt_, ptName_, ptSize_: Medium, offset__: {-5, -5}] := 
 Module[{},
  GraphicsGroup[{{PointSize[ptSize], Point[pt]}, 
    Text[ptName, Offset[offset, pt]]}]
    ]

and do simple calls like

Graphics[{namedPoint[{0, 0}, "O"], namedPoint[{1, 1}, "A"], 
  namedPoint[{2, -1}, "B"]}, ImageSize -> 100]

which gives and image like this

Another possibility is to use Callout inside of ListPlot:

ListPlot[{Callout[p0[[1]], "A"], Callout[p1[[1]],"B"]}]