Let $A=\begin{bmatrix} 1 & 2\\ 3& 4 \end{bmatrix}$ then det$(A^3-6A^2+5A+3I)=3$

Yes, this looks fine. Since $A$ satisfies its own characteristic polynomial, you have: $$\color{blue}{A^2-5A-2I=O}$$ and so, as you wrote: $$A^3-6A^2+5A+3I=\underbrace{\left(\color{blue}{A^2-5A-2I}\right)}_{\color{blue}{O}}\left(A-I\right)+2A+I=2A+I$$ which leaves you with the (easier) $\det\left(2A+I\right)$ and that is indeed $3$.