Why does Mathematica try to take the first element of the empty list when plotting?

I am using V12.1.1.0 on MacOS 10.13.4 and I can reproduces your problem. I found the following work-around:

ListPlot[
  {Callout[
     {{1500, 4}, {1510, 4}, {1520, 4}, {1850, 6}}, 
       "a", {{2500, 6}, {0, .5}}, {2100, 6}],
   Callout[{{1500, 1}, {18100, 1}}, "b"]},
   Joined -> True, PlotRange -> All]

The work-around requires giving up the easy-to-use PlotLabels option and placing a callout wrapper around the 1st dataset, so the excellent micromanaging capabilities provided by Callout can be used.

As a workaround

$Version

"12.1.1 for Mac OS X x86 (64-bit) (June 19, 2020)"

data = {{{1500, 4}, {1510, 4}, {1520, 4}, {1850, 6}}, {{1500, 1}, {18100, 1}}};

labels = {"a", "b"};

Show[
 ListPlot[#[[1]],
    PlotStyle -> ColorData[97][#[[2]]],
    PlotLabels -> #[[3]],
    Joined -> True] & /@
  Transpose[{data, Range[Length[data]], labels}],
 PlotRange -> All,
 AxesOrigin -> {0, 0}]

Alternatively,

Show[
 ListPlot[#1, 
    PlotStyle -> ColorData[97][#2], 
    PlotLabels -> #3, 
    Joined -> True] & @@@ 
  Transpose[{data, Range[Length[data]], labels}], 
 PlotRange -> All, 
 AxesOrigin -> {0, 0}]