Derivative of quadratic matrix form with respect to the matrix

$$ f(M)=\sum_{i,j}x_iM_{i,j}x_j. $$ Therefore $$ \frac{\partial f(M)}{\partial M_{i,j}} = x_ix_j = (xx^T)_{i,j}. $$

The strategy is to write the expression as a scalar using index notation, take the derivative, and re-write in matrix form.

Note that to write the function as a summation of matrices we have to write just one scalar as a matrix multiplication because the function is scalar: $$ f(M)= [x^TMx]_{11}= \sum_i x_{i1}[Mx]_{i1} $$ At first summation, we have $x_{i1}$ because we have $x^T$ as the first term.

Now expand $[Mx]_{i1}$ $$ f(M)= [x^TMx]_{11}= \sum_i x_{i1}\sum_j M_{ij}x_{j1}=\sum_i \sum_j x_{i1} M_{ij}x_{j1} $$

Now take the derivative with respect to $M_{ij}$

$$ \frac{\partial f(M)}{\partial M_{ij}}=\sum_i \sum_j x_{i1}x_{j1} $$

Looking at the indices, we can see that

$$ \sum_i \sum_j x_{i1}x_{j1}=\sum_i \sum_j x_{j1}x_{i1}=[xx^T]_{ji} $$

Therefore,

$$ \frac{\partial f(M)}{\partial M_{ij}}=xx^T $$