Factoring the square of this polynomial?

Try

Simplify[1 - (2 b^2)/a^2 + b^4/a^4 /. a ->  b/λ] /. λ -> b/a
(*(-1 + b^2/a^2)^2*)

Explanation of the problem

The reason it never reaches the form you want despite of being minimal by LeafCount is that that form is never tried, so its LeafCount is never measured nor compared. Look at the attempts with a fresh kernel (as Simplify is cashed).

So the solution is to find a way for Simplify to explore the form you want using TransformationFunctions and score it as optimal as per your definition ComplexityFunction -> LeafCount.

{sol, {attempts}} = Reap@Simplify[
   1 - (2 b^2)/a^2 + b^4/a^4
   , ComplexityFunction -> ((Sow[#]; LeafCount[#]) &)
   ];

enter image description here

Solution requested: Simplify

We use TransformationFunctions to tell Simplify to explore the form you want, and ComplexityFunction to define how these alternatives are scored.

One way is to use the CompleteSquare described below

Simplify[
 1 - (2 b^2)/a^2 + b^4/a^4
 , TransformationFunctions -> {Automatic, CompleteSquare[#, b^2] &}
 , ComplexityFunction -> LeafCount
 ]

(-1 + b^2/a^2)^2

enter image description here

Or based on the answer by Ulrich Neumann, also

transf[expr_] := Module[
  {vars, tvar},
  vars = Variables[expr];
  ReplaceAll[
   Simplify@ReplaceAll[expr, vars[[1]] -> vars[[2]]/tvar]
   , tvar -> Divide @@ vars[[{2, 1}]]
   ]
  ]

Simplify[
 1 - (2 b^2)/a^2 + b^4/a^4
 , TransformationFunctions -> {Automatic, transf}
 , ComplexityFunction -> LeafCount
 ]
(* (-1 + b^2/a^2)^2 *)

PolynomialForm

The undocumented PolynomialForm allows placing the terms in the desired order with the option TraditionalOrder->True.

PolynomialForm[(-1+b^2/a^2)^2,TraditionalOrder->True]
(* (b^2/a^2-1)^2 *)

enter image description here

or

Format[bda] = DisplayForm@FractionBox["b", "a"];

PolynomialForm[
 (-1 + b^2/a^2)^2 /. b -> bda a
 , TraditionalOrder -> True
 ]

enter image description here

Alternative Solution: CompleteSquare

Other people have suggested way to complete the square. You can force that form by

CompleteSquare[f_, x_] := Module[
  {a, b, c},
  {c, b, a} = CoefficientList[f, x];
  Assuming[
   Sqrt[a] > 0,
    (FullSimplify[Sqrt[a] x] + 
       FullSimplify[b/(2 Sqrt[a])])^2 + (FullSimplify[(a c - b^2/4)])
   ]]

CompleteSquare[1 - (2 b^2)/a^2 + b^4/a^4, b^2]
(* (-1 + b^2/a^2)^2 *)

You can do

Factor[1 - (2 b^2)/a^2 + b^4/a^4] // FullSimplify

or (as suggested by @Thies Heidecke)

FullSimplify[1 - (2 b^2)/a^2 + b^4/a^4]

with identical output: $$\frac{\left(a^2-b^2\right)^2}{a^4}$$

which is close.