Writing a program that will filter the integer solutions

You can use Solve. Your equation:

eqn[r_] := b == (-12 + Sqrt[3] Sqrt[3 (-4 + r)^2 + 12 a^2 (-2 + r) - 4 a (-5 + r) (-2 + r) + 4 a^3 (-2 + r)^2] + 3 r)/(6 (-2 + r))

Using Solve (with $r=4$):

Solve[eqn[4] && 1 < a < 100, {a, b}, Integers]

{{a -> 24, b -> 70}}

Using Solve (with $r=11$):

Solve[eqn[11] && 1 < a < 100, {a, b}, Integers]

{{a -> 25, b -> 73}}

You might try IntegerQ

Table[b = (-12 +Sqrt[3] Sqrt[3 (-4 + r)^2 + 12 a^2 (-2 + r) - 4 a (-5 + r) (-2+ r) +4 a^3 (-2 + r)^2] + 3 r)/(6 (-2 + r)) /. r -> 4;
If[IntegerQ[b], {b, a}, Nothing], {a, 2, 50}]
(*{{70, 24}}*)

You can use Cases here:

b[r_, a_] := (-12 + Sqrt[3] Sqrt[3 (-4 + r)^2 + 12 a^2 (-2 + r) - 4 a (-5 + r) (-2 + r) + 4 a^3 (-2 + r)^2] + 3 r)/(6 (-2 + r));

With r=4

Cases[Table[b[4, a], {a, 2, 100}], _Integer]

{70}

If you want to retrieve {a,b} pairs:

Cases[Table[{a, b[4, a]}, {a, 2, 100}], {_, _Integer}]

{{24, 70}}

It's interesting to look at {a,b} points for a bunch of r values:

t = Cases[
       Table[Cases[Table[{a, b[r, a]}, {a, 2, 100}], {_, _Integer}], {r, 3, 20}], Except[{}]];

ListPlot[t, AxesLabel -> {a, b}]