On the definition of the associated Legendre polynomials

Yes, they have different definitions:

• Maple

• Mathematica

• Wikipedia

$$P_\ell^m(x)=(-1)^m \left(1-x^2\right)^{m/2} \frac{\mathrm d^m}{\mathrm dx^m} P_\ell(x)$$

This difference produces different complex phases (different branches of the square root).

Something ybeltukov forgot to mention: LegendreP[] takes a third optional parameter corresponding to the "type" of Legendre function needed. There are three types, all agreeing within the unit disk, but having different branch cut structures outside it. By default, the type 1 function is computed: LegendreP[n, m, z] == LegendreP[n, m, 1, z].

In particular, Maple's choice of type corresponds to type 3 in Mathematica's convention. Thus:

Table[(-1)^n LegendreP[n, -1, 3, -3]/Sqrt[2], {n, 5}]
   {1, 3, 11, 45, 197}

P.S. Similar remarks hold for LegendreQ[].