Abstract Smooth Manifolds vs Embedded Smooth Manifolds

There's a reason that definition does not require that the map $\phi$ in a chart $(U,\phi)$ be a diffeomorphism: that would require knowing already that $M$ is a smooth manifold, but since that is what is being defined, the definition would become circular.

However, once a smooth manifold $(M,\mathcal{A})$ is defined, then one can move forward and define smooth functions on open subsets of $M$. Namely, for each open set $W \subset M$, a function $\xi : W \to \mathbb{R}^k$ is smooth if and only if for each chart $(U,\phi)$ in the atlas $\mathcal{A}$ the map $\xi \circ \phi^{-1} : \phi(W \cap U) \to \mathbb{R}^k$ is smooth. And then, by applying the definition of a smooth atlas, it is now an easy lemma to prove that if $(U,\phi)$ is a chart in the atlas $\mathcal{A}$ then $\phi : U \to \mathbb{R}^m$ is indeed smooth.

It wouldn't make sense for the definition to require $\phi$ and $\psi$ to be diffeomorphisms, because you can't define what it means for them to be diffeomorphisms until you already have a smooth structure on $M$. This is in contrast with the situation for embedded manifolds, where you can define what it means for a map to be smooth using the usual notion of differentiation of maps between subsets of $\mathbb{R}^n$.

That said, any chart of a smooth manifold is a diffeomorphism. This is pretty much immediate from the definitions: for a map between manifolds to be smooth, that means its compositions with charts give smooth maps between open subsets of $\mathbb{R}^n$. But in the case that your map is itself a chart (or the inverse of a chart), these compositions are exactly the maps of the form $\psi \circ \phi^{-1}$ which the definition requires to be diffeomorphisms (in particular, smooth).

The point is that since your manifold is abstractly defined and not embedded, it's not completely obvious what you mean by diffeomorphic (before you define the smooth structure!).

In the case of an embedded submanifold, you can just ask that $\phi$ and $\psi$ extend to diffeomorphisms defined on open subsets of the ambient space, but in the second definition you cannot do that. So what you do is you define a map $f: M \to \mathbb{R}^n$ to be smooth if on every chart domain $(U,\phi)$, the map $f \circ \phi^{-1}$ is smooth.

To be able to do that consistently, you need the condition that the transition maps are smooth, and a posteriori, yes, the charts are smooth for the smooth structure they induce.