Order of Cyclic Subgroups

The book is correct; your professor is correct when $d$ divides (i.e. goes into) $n$.

Note that for any $g\in G$, $$|\langle g\rangle|=\text{ord}(g).$$ The general relation is that, if $\text{ord}(a)=n$, then $$\text{ord}(a^d)=\frac{n}{\gcd(d,n)}.$$ For example, consider the cyclic group $G=\mathbb{Z}/6\mathbb{Z}=\{0,1,2,3,4,5\}$ with operation $+\,$, and let $a=1$. It has order $n=6$. Let $d=5$; then $a^5$ means $$a+a+a+a+a=5$$ so $H=\langle a^5\rangle=\langle 5\rangle=G$, so $|H|=6=\frac{6}{\gcd(5,6)}$. In contrast, the statement that $|H|=\frac{6}{5}$ doesn't even make any sense.

---

If you intended $n$ to be the order of $G$, then both answers will be false unless $G$ is a cyclic group and $\langle a\rangle=G$ (in my answer above, I assumed you meant $n=\text{ord}(a)$, which agrees in this case, i.e. $\text{ord}(a)=n=|G|$ if and only if $G$ is cyclic and $G=\langle a\rangle$). For example, in the group $$G=(\mathbb{Z}/6\mathbb{Z})\times(\mathbb{Z}/2\mathbb{Z}),$$ for which $|G|=12$, the element $a=(1,1)$ has order 6, and $$a^5=(1,1)+(1,1)+(1,1)+(1,1)+(1,1)=(5,1),$$ hence $$H=\langle a^5\rangle=\langle(5,1)\rangle$$ has $|H|=6$, but $$|H|\neq\frac{12}{1}=\frac{12}{\gcd(5,12)}$$ and $$|H|\neq\frac{12}{5}$$