What is the coefficient of $x^3y^5$ in the expansion of $(1+xy+y^2)^n$?

What is the coefficient of $x^3y^5$ in the expansion of $(1+xy+y^2)^n$?

Question

Let $n$ be a positive integer. The coefficient of $x^3y^5$ in the expansion of

$$(1+xy+y^2)^n$$

equals

$$(a) \quad n,\quad(b) \quad 2^n,\quad(c) \quad \begin{pmatrix}n\\3\end{pmatrix} \begin{pmatrix}n\\5\end{pmatrix},\quad(d) \quad 4\begin{pmatrix}n\\4\end{pmatrix},\quad(e) \quad\begin{pmatrix}n\\8\end{pmatrix}.$$

Suggestion

Could we write $(1+xy+y^2)^n$ as $(1+y(x+y))^n$? What happens if we expand this with the binomial theorem now?

Solution

We can write the expression as

$$(1+y(x+y))^n = 1 + \begin{pmatrix}n\\1\end{pmatrix}y(x+y)+\begin{pmatrix}n\\2\end{pmatrix}y^2(x+y)^2+\begin{pmatrix}n\\3\end{pmatrix}y^3(x+y)^3+\begin{pmatrix}n\\4\end{pmatrix}y^4(x+y)^4+\begin{pmatrix}n\\5\end{pmatrix}y^5(x+y)^5+...$$

The powers in $x^3y^5$ add to 8, so the only term that interests us is $\begin{pmatrix}n\\4\end{pmatrix}y^4(x+y)^4.$

We need the $x^3y$ term in $(x+y)^4$, which has coefficient $\begin{pmatrix}4\\3\end{pmatrix} = 4$. Thus the coefficient of $x^3y^5$ overall is $4\begin{pmatrix}n\\4\end{pmatrix}$, and the answer is (d).