A manifold with $\pi_1 = \pi_2 = \pi_4 = 0, \pi_3 = \mathbb{Z}$
There is actually an $8$ dimensional manifold which has your desired homotopy groups: The Lie group $SU(3)$.
In fact, the result follows from Bott Periodicty. More specifically, for $U = \bigcup_n U(n)$, the union of the unitary groups, Bott periodicity shows that $\pi_k(U)$ only depends on the parity of $k$: when $k$ is even, this group is isormphic to $\mathbb{Z}$, and when $k$ is odd it is trivial.
But from the inclusion $U(3)\subseteq U$ is at least 5 connected, so $\pi_k(U(3))\cong \pi_k(U)$ for $k\leq 4$. Finally, since $SU(3)\times S^1\cong U(3)$, $SU(3)$ must have the right homotopy groups.
Consider the $n$-ball for $n$ large. Attach a 3-handle and call the resulting manifold $M$. It should be relatively easy to see the map $\pi_4(\partial M) \to \pi_4(M)$ is surjective (one way is by turning the cobordism "upside down" and thinking about the 3-handle as an n-3 handle and the 0-handle as an n-handle). Now attach 5-handles to the boundary along all the generators of $\pi_4(\partial M)$ (which can be represented by smoothly framed embeddings by Whitney when $n$ is large enough).
Now, the resulting manifold with boundary has a handle decomposition (Morse function) with one 0-handle, one 3-handle and one 5-handle (its easy to see the $\pi_4$ you are killing is actually $\Bbb Z/2$), so its double has one 0-handle, one 3-handle, one 5-handle, one $n$-5 handle, one $n$-3 handle and one $n$ handle. For $n$ large enough, only the first three handles will affect the desired homotopy groups.
This is a typical argument to construct compact smooth manifolds with any $k$-type you want. For your case the smallest $n$ where this argument works is $10$.
Let $G$ be a simply connected compact Lie group*. There is a theorem of Bott, see Theorem $21.7$ of Milnor's Morse Theory for example, that shows that $\Omega G$ has the homotopy type of a CW complex with no odd-dimensional cells, and only finitely many cells of each even dimension. It follows that $\pi_2(G) = 0$ and $\pi_3(G)$ is free of finite rank, i.e. $\pi_3(G) \cong \mathbb{Z}^r$ for some $r$. With a little more work, see Bott's An application of the Morse theory to the topology of Lie-groups, one can show that if $G$ is simple, then $\pi_3(G) \cong \mathbb{Z}$.
What about $\pi_4(G)$ for $G$ simple? Theorem IV of Bott & Samelson's Applications of the Theory of Morse to Symmetric Spaces states that there are only two possibilities: $0$ and $\mathbb{Z}_2$. Furthermore, they show that $\pi_4(G) = \mathbb{Z}_2$ if and only if $G = SU(2) = \operatorname{Spin}(3) = Sp(1)$, or $G = Sp(n)$ for $n \geq 2$. By the classification of simple Lie groups, we see that $\pi_4(G) = 0$ if and only if $G$ is one of $SU(n)$ for $n \geq 3$, $\operatorname{Spin}(n)$ for $n \geq 5$, $E_6$, $E_7$, $E_8$, $F_4$, or $G_2$. All of these Lie groups are examples of closed, connected, smooth manifolds with the desired homotopy groups.
*Although it's not needed for your purposes, let me just emphasise that these results hold for all Lie groups. The simply connectedness hypothesis can be removed by passing to the universal cover, and the compactness hypothesis can be removed because every Lie group deformation retracts onto a compact subgroup.