when mapping cone is contractible

Ok, going off my second comment from above, in exercise 9 from section 4.2 Hatcher gives a hint that solves your problem. Let $X$ be an acyclic CW-complex which isn’t contractible (I'll give an example below to be complete). Let $f: X \rightarrow *$. The mapping cone of this is $SX$, the suspension of $X$. Exercise 8 of section 4.2 proves that $SX$ is contractible because $X$ is acyclic. But the map $X \rightarrow *$ is not a homotopy equivalence.

Solution to Exercise 8: Suppose $\tilde{H}_*(X)=0$. From $H_0$ we see $X$ is path connected, so $SX$ is simply connected. Thus $H_* (SX) = 0 $ . By the Hurewicz theorem $\pi_*(SX)=0$ so $SX$ is contractible as desired.

An acyclic CW-complex which is not contractible (thanks to link)... Let $a$ and $b$ be the two loops in $X=S^1 \vee S^1$. Glue in two 2-cells along the words $a^5b^{-3}$ and $b^3(ab)^{-2}$. Then $\pi_1(X) = \langle a,b|a^5b^{-3},b^3(ab)^{-2}\rangle$ and this surjects onto the symmetry group of a regular dodecahedron (so $\pi_1(X)\not \cong 0$) while $H_1(X)$ is trivial because it's the abelianization and playing with symbols lets you reduce those relations to $H_1(X) \cong \langle a,b | a,b\rangle \cong 0$. It's clear that $H_n(X)\cong 0$ for $n>1$ because of the dimensions of the cells involved.

The accepted answer addresses the version of the question for topological spaces. But there is also a positive result for triangulated categories: a morphism $f \colon X \to Y$ is an isomorphism if and only if any distinguished triangle $X \to Y \to Z \to X[1]$ has $Z \cong 0$. See Tag 05QR.

This in particular applies to the homotopy category $K(\mathcal A)$ of chain complexes with values in an additive category $\mathcal A$, which is a triangulated category by Tag 014S. Applying this to the mapping cone in $K(\mathcal A)$ shows that $C(f)$ is contractible if and only if $f$ is a homotopy equivalence.

This also applies to the stable homotopy category of spectra, where it implies that a map of topological spaces $f \colon X \to Y$ with contractible mapping cone is a stable homotopy equivalence.

In examples like David White's answer, you only need to go up to the first suspension $\Sigma X$, by construction (but see also this answer). I'm not sure if something like this is supposed to be true in general, but I'm guessing it's more complicated.