When A and B are of different order given the $\det(AB)$,then calculate $\det(BA)$

You will always get $\det(BA)=0$. The reason for this is very simple : $BA$ is a $3\times 3$ matrix with rank at most $2$; thus it is not invertible and thus it has $0$ determinant.

One possible definition of the rank is that it is the dimension of the subspace generated by the columns or the lines of the matrices. The lines of $BA$ are obtained by taking linear combinations of the lines of $A$ : for example, if $$B=\begin{pmatrix}b_{11} & b_{12} \\ b_{21} & b_{22} \\ b_{31}& b_{32} \end{pmatrix},\ A= \begin{pmatrix}a_{11} & a_{12} & a_{13}\\ a_{21} & a_{22} & a_{23} \end{pmatrix},$$ then the first line of $BA$ is $$b_{11}\cdot \begin{pmatrix}a_{11} & a_{12} & a_{13} \end{pmatrix} + b_{12}\cdot \begin{pmatrix} a_{21} & a_{22} & a_{23} \end{pmatrix}.$$

Since the lines of $BA$ are generated by only two lines (of $A$), the dimension of the generated subspace cannot be more than $2$.

This also means that the lines of $BA$ cannot be linearly independent; thus there must be some linear relation between them, which means that when you convert it to echelon matrix you will certainly get a line of $0$s.

I think what you are looking for is an answer like this

$B=\begin{pmatrix}b_{11} & b_{12} \\ b_{21} & b_{22} \\ b_{31}& b_{32} \end{pmatrix}$, $A= \begin{pmatrix}a_{11} & a_{12} & a_{13}\\ a_{21} & a_{22} & a_{23} \end{pmatrix}$,

Then \begin{align}BA & =\begin{pmatrix}b_{11}a_{11}+b_{12}a_{21} & b_{11}a_{12}+b_{12}a_{22} & b_{11}a_{13}+b_{12}a_{23}\\ b_{21}a_{11}+b_{22}a_{21} & b_{21}a_{12}+b_{22}a_{22} &b_{21}a_{13}+b_{22}a_{23} \\b_{31}a_{11}+b_{32}a_{21} & b_{31}a_{12}+b_{32}a_{22} & b_{31}a_{13}+b_{32}a_{23} \end{pmatrix} \\ & =\begin{pmatrix}b_{11}a_{11}+b_{12}a_{21}+0.0 & b_{11}a_{12}+b_{12}a_{22}+0.0 & b_{11}a_{13}+b_{12}a_{23}+0.0\\ b_{21}a_{11}+b_{22}a_{21}+0.0 & b_{21}a_{12}+b_{22}a_{22}+0.0 &b_{21}a_{13}+b_{22}a_{23}+0.0 \\b_{31}a_{11}+b_{32}a_{21}+0.0 &b_{31}a_{12}+b_{32}a_{22}+0.0 & b_{31}a_{13}+b_{32}a_{23} +0.0 \end{pmatrix}\\ & =\begin{pmatrix}b_{11} & b_{12} & 0\\ b_{21} & b_{22} &0 \\b_{31} & b_{32} & 0 \end{pmatrix} \begin{pmatrix}a_{11} & a_{12} & a_{13}\\ a_{21} & a_{22} &a_{23} \\0 & 0 & 0 \end{pmatrix}.\end{align}

Let $C=\begin{pmatrix}b_{11} & b_{12} & 0\\ b_{21} & b_{22} &0 \\b_{31} & b_{32} & 0 \end{pmatrix}$ and $D= \begin{pmatrix}a_{11} & a_{12} & a_{13}\\ a_{21} & a_{22} &a_{23} \\0 & 0 & 0 \end{pmatrix}$.

Now, $BA=CD$. Therefore, $\det(BA)=\det(CD)=\det(C)\cdot \det(D)=0\cdot 0=0$.

Let $ T = BA $, then $ T : \mathbb{R}^3 \to \mathbb{R}^3 $ is a linear transformation with nontrivial kernel, as $ A : \mathbb{R}^3 \to \mathbb{R}^2 $ cannot be an injection. Thus, it is not invertible, and $ \det(T) = 0 $.