Finding coefficient of polynomial?

It's basically the number of ways you can write $12$ as a sum of three integers all greater than or equal to $3$, where order matters. So $12=3+3+6$, $12=3+4+5$, $12=4+4+4$. The first can be rearranged three ways, the second can be rearranged six ways and the last can only be arranged one way. So yes, I believe the answer is $10$.


$$(x^3+x^4+x^5+x^6+\cdots)^3=x^9(1+x+x^2+\cdots)^3=x^9\left(\dfrac1{1-x}\right)^3=x^9(1-x)^{-3}$$

Now, we need the coefficient of $x^3$ in $(1-x)^{-3}$

Now the $r+1,(r\ge0)$th term of $(1-x)^{-3}$ is $$\dfrac{-3(-4)(-5)\cdots(-r)(-r-1)(-r-2)}{1\cdot2\cdot3\cdot r}(-x)^r=\binom{r+2}2x^r$$


From the OP, the coefficient of $x^{12}$ in $(x^3 + x^4 + x^5 + x^6 + \cdots)^3$ is equal to that of $x^3$ in $(1+x+x^2+x^3)^3$. This is equivalent to asking the number of ways to pick one $x^i$ below in each row so that the product of the $x^i$ picked in each row is of the form $kx^3$ for some number $k$.

$$ \require{enclose} \bbox[border:2px solid red] { \begin{array}{c|c|c|c} x^0 & x^1 & x^2 & x^3 \\ \hline x^0 & x^1 & x^2 & x^3 \\ \hline x^0 & x^1 & x^2 & x^3 \end{array} } $$

If we focus on indices, we will find out that this is equivalent to asking the number of ways of choosing one number from each row so that the sum of three chosen numbers adds up to three.

$$ \bbox[border:2px solid red] { \begin{array}{c|c|c|c} 0 & 1 & 2 & 3 \\ \hline 0 & 1 & 2 & 3 \\ \hline 0 & 1 & 2 & 3 \end{array} } $$

Therefore, the problem is asking for $$\#\{x,y,z\in\Bbb Z_0^+ \mid x \color{blue}{\fbox+} y \color{red}{\fbox+} z = 3\}.$$

Hence, the answer is very simple: ${5 \choose 2} = 10$. First, imagine that we have a $5\times 1$ grid.

\begin{array}{|l|l|l|l|l|} \hline \\ &&&& \\ \hline \end{array}

Then you choose two grids to put $\color{blue}{\fbox+}$ and $\color{red}{\fbox+}$. These two plus signs symbolises $x\color{blue}{\fbox+}y\color{red}{\fbox+}z=3$. Therefore, $\color{blue}{\fbox+}$ should be at the left of $\color{red}{\fbox+}$. The picture below serves as an example.

\begin{array}{|l|l|l|l|l|} \hline \\ &\color{blue}{\fbox+}&\color{red}{\fbox+}&& \\ \hline \tag{*} \label{*} \end{array}

Fill the remaining grids with three $\enclose{circle}{1}$ to see what happens.

\begin{array}{|l|l|l|l|l|} \hline \\ \enclose{circle}{1}&\color{blue}{\fbox+}&\color{red}{\fbox+}&\enclose{circle}{1}&\enclose{circle}{1} \\ \hline \end{array}

Therefore, this example shows one possibility $x=1,y=0,z=2$. You may make up others by choosing other combinations in \eqref{*}.