A metacyclic group $G$ and its Sylow $p$-subgroup, where $p$ is the smallest prime dividing the order of $G$

Here is my comment expanded slightly. By factoring out Sylow $q$-subgroups of $G'$ for $q \ne p$, we can assume that $G'$ is a $p$-group, so $G' \cap P = G'$. Hence you have shown (using the fact that $p$ is the smallest prime dividing $|G|$) that $G = PC_G(G')$.

Let $Q$ be a Sylow $p$-complement of $C_G(G')$. So $Q$ is also a Sylow $p$-complement of $G$, and $G=PQ$. Since $Q \cap G' = 1$, $Q$ must be abelian (in fact cyclic).

Since $G/G'$ is abelian, $Q$ centralizes $P/G'$ and $Q$ centralizes $P'$ from its definition. Now any automorphism of a finite $p$-group $P$ that centralizes a normal subgroup $N$ of $P$ and induces the identity on $P/N$ must have order a power of $p$. This is a standard result, and is not hard to prove. So in fact $Q$ centralizes $P$, and hence $Q \le Z(G)$.

So $G' = (PQ)' = P'$, and hence $P/P'$ is cyclic, which imples that $P$ is cyclic.