Calculating maximum power transfer for given circuit

Since the unknown resistance, \$R\$, and the fixed voltage supply are in series, they can be swapped without loss for analysis. The resulting schematic is as simple as the following case:

^{simulate this circuit – Schematic created using CircuitLab}

You only have one node to solve for, making this almost trivial:

$$\frac{V}{R_1=5\:\Omega}+\frac{V}{R}=100\:\text{mS}\cdot\left(V-{-5}\:\text{V}\right)+\frac{-5\:\text{V}}{R}$$

Solve the above equation for \$V\$ and then apply it to the power equation: \$P_R=\frac{\left(V\,-\,{-5}\:\text{V}\right)^2}{R}\$. Take the differential with respect to \$R\$ and then set that equal to zero and solve. You will find the solution to be \$R=10\:\Omega\$.

In sympy:

v=Symbol('v')
r=Symbol('r')
solve(Eq(derivative(simplify((solve(Eq(v/5+v/r,.1*(v+5)-5/r),v)[0]+5)**2/r),r),0),r)[0]

10.0000000000000

First, I will present a method that uses Mathematica to solve this problem. When I was studying this stuff I used the method all the time (without using Mathematica of course). Besides that the answer of @jonk is excellent.

Well, we are trying to analyze the following circuit:

^{simulate this circuit – Schematic created using CircuitLab}

When we use and apply KCL, we can write the following set of equations:

$$\text{n}\cdot\text{V}_1=\text{I}_1+\text{I}_2\tag1$$

When we use and apply Ohm's law, we can write the following set of equations:

$$ \begin{cases} \text{I}_1=\frac{\text{V}_1}{\text{R}_1}\\ \\ \text{I}_2=\frac{\text{V}_2}{\text{R}_2} \end{cases}\tag2 $$

We also know that \$\text{V}_1-\text{V}_2=\text{V}_\text{i}\$.

Now, we can solve for \$\text{V}_1\$:

$$\text{V}_1=\frac{\text{R}_1\text{V}_\text{i}}{\text{R}_1+\text{R}_2\left(1-\text{n}\text{R}_1\right)}\tag3$$

Where I used the following Mathematica-code to find \$(3)\$:

In[1]:=Clear["Global`*"];
FullSimplify[
 Solve[{n*V1 == I1 + I2, I1 == V1/R1, I2 == V2/R2, 
   V1 - V2 == Vi}, {V1, V2, I1, I2}]]

Out[1]={{V1 -> (R1 Vi)/(R1 + R2 - n R1 R2), 
  V2 -> -(((1 - n R1) R2 Vi)/(R1 + R2 - n R1 R2)), 
  I1 -> Vi/(R1 + R2 - n R1 R2), 
  I2 -> -((Vi - n R1 Vi)/(R1 + R2 - n R1 R2))}}

So, the power in the resistor \$\text{R}_1\$ is given by:

$$\text{P}_{\text{R}_1}=\frac{\text{V}_1^2}{\text{R}_1}=\left(\frac{\text{R}_1\text{V}_\text{i}}{\text{R}_1+\text{R}_2\left(1-\text{n}\text{R}_1\right)}\right)^2\cdot\frac{1}{\text{R}_1}=\text{R}_1\cdot\left(\frac{\text{V}_\text{i}}{\text{R}_1+\text{R}_2\left(1-\text{n}\text{R}_1\right)}\right)^2\tag4$$

In order to find the maximum, we can use:

$$\frac{\partial\text{P}_{\text{R}_1}}{\partial\text{R}_1}=0\space\Longleftrightarrow\space\left(\text{R}_2+\text{R}_1\left(\text{nR}_2-1\right)\right)\text{V}_\text{i}^2=0\space\Longleftrightarrow\space\text{R}_1=\frac{\text{R}_2}{1-\text{n}\text{R}_2}\tag5$$

Where I used the following Mathematica-code to find \$(5)\$:

In[2]:=Clear["Global`*"];
FullSimplify[Solve[{D[R1 *(Vi/(R1 + R2 - n R1 R2))^2, R1] == 0}, R1]]

Out[2]={{R1 -> R2/(1 - n R2)}}

Using your values, we get:

$$\text{R}_1=\frac{5}{1-\frac{1}{10}\cdot5}=10\space\Omega\tag6$$