Replacing one element of list

Let's say your matrix is A, this method does not depend of length of inner lists:

A[[All, All, -1]] = A[[All, All, -1]] /. {x -> y, y -> x};
A

This one doesn't even care about the depth of the array:

A /. a : {(x | y) ..} :> MapAt[# /. {x -> y, y -> x} &, a, {-1}]

MapAt

dt = RandomChoice[{x, y}, {2, 3, 4}];
dt //TeXForm

$\left( \begin{array}{ccc} \{x,y,y,y\} & \{y,x,y,x\} & \{x,x,x,y\} \\ \{y,x,x,x\} & \{y,y,x,y\} & \{x,x,y,y\} \\ \end{array} \right)$

f1 = MapAt[# /. {x -> y, y -> x} &, #, {{All, All, -1}}] &;
f1 @ dt // TeXForm

$\left( \begin{array}{ccc} \{x,y,y,x\} & \{y,x,y,y\} & \{x,x,x,x\} \\ \{y,x,x,y\} & \{y,y,x,x\} & \{x,x,y,x\} \\ \end{array} \right)$

Replace

f2 = Replace[#, {a__, b : x | y} :> {a, b /. {x -> y, y -> x}}, Infinity] &;
f2 @ dt // TeXForm

$\left( \begin{array}{ccc} \{x,y,y,x\} & \{y,x,y,y\} & \{x,x,x,x\} \\ \{y,x,x,y\} & \{y,y,x,x\} & \{x,x,y,x\} \\ \end{array} \right)$