Determine whether a list is a reordering of another

You can declare prod Orderless, and then use Simplify to obtain the result:

SetAttributes[prod, Orderless]
c1 prod[P1, P2, state] + c2 prod[P2, P1, state] // Simplify

(c1 + c2) prod[P1, P2, state]

I'm not sure I follow your idea for how to extract the coefficient, but perhaps something like this:

prod /: Times[a_prod, b_prod] := If[a === b, 1, 0]
Sqrt[sum sum]

Sqrt[(c1+c2)^2]

Also see the comment by Michael E2 on how to make sure that state is not mixed up with the orderless parameters.

(c1 prod[P1, P2, state] + c2 prod[P2, P1, state]) /. p_prod :> Sort[p] // Simplify

  (c1 + c2) prod[P1, P2, state]

If prod is orderless only in the first two arguments, you can do:

(c1 prod[P1, P2, state] + c2 prod[P2, P1, state]) /. 
  prod[a_, b_, st_] :> prod[## & @@ Sort[{a, b}], st] // Simplify

(c1 + c2) prod[P1, P2, state]

(c1 prod[P1, P2, state] + c2 prod[P2, P1, state]) /. 
  prod[OrderlessPatternSequence[a_, b_], st_] :> prod[a, b, st] // Simplify

(c1 + c2) prod[P1, P2, state]

Related