Is $W^2-t$ martingale?

Perhaps you just made a simple substitution mistake. When you wrote $$(dW_t)^2 = (dW - t)^2,$$ you actually substituted what you thought would be $dX_t$. But in fact, $X_t = f(t, W_t)$, and so the term $(dX)^2$ in your Ito formula corresponds to just $(dW_t)^2 = t$, as in the answer you linked in the comments. Indeed, there $f$ was a function of $S_t$ alone, and an expression for $dS_t$ was given. In your situation, you have an expression in terms of $X_t = W_t^2 - t$ and it's just a question of evaluating $dX_t$.