Finding Maximum Area of a Rectangle in an Ellipse

Its easier to solve this question using parametric points.

Let one vertex of the rectangle be $(a\cos\theta,b\sin\theta)$.

The other vertices are $(a\cos\theta,-b\sin\theta)$, $(-a\cos\theta,b\sin\theta)$, $(-a\cos\theta,-b\sin\theta)$

The area of rectangle formed is $$A(\theta)=4ab\cos\theta\sin\theta=2ab\sin2\theta$$

Maximum area is $2ab$ and it occurs when $\theta=\frac{\pi}{4}$ (or when $\sin2\theta$ is maximum).

When $\theta=\frac{\pi}{4}$, $x$-coordinate $=a\cos\frac{\pi}{4}=\frac{a}{\sqrt{2}}$

Your mistake is here $$ A'(x) = 4(\sqrt{ b^2 - \frac{b^2x^2}{a^2}}) +\color{red}{\frac12} \times4x\left( (b^2 - \frac{b^2x^2}{a^2})^{\frac{-1}{2}} \times \frac{-2b^2x}{a^2} \right). $$

Let equation of ellipse be

$$ \frac{x^2}{a^2} + \frac{y^2}{b^2} = 1$$

Solving for y

$$ y = \sqrt{ b^2 - \frac{b^2x^2}{a^2}} $$

Let area of a rectangle be $4xy$

$$ A = 4xy $$

$$ A = 4x(\sqrt{ b^2 - \frac{b^2x^2}{a^2}}) $$

$$ A'(x) = 4(\sqrt{ b^2 - \frac{b^2x^2}{a^2}}) + 4x\left( (b^2 - \frac{b^2x^2}{a^2})^{\frac{-1}{2}} \times \frac{-b^2x}{a^2} \right) $$

$$ A'(x) = 4\sqrt{ b^2 - \frac{b^2x^2}{a^2}} + \frac{-4x^2b^2}{\sqrt{ b^2 - \frac{b^2x^2}{a^2}}a^2} = 0 $$

$$ 4a^2\left(b^2 - \frac{b^2x^2}{a^2} \right) - 4x^2b^2 = 0 , \sqrt{ b^2 - \frac{b^2x^2}{a^2}a^2} \neq 0 $$

$$ 4a^2\left(b^2 - \frac{b^2x^2}{a^2} \right) - 4x^2b^2 = 0 $$

$$ 4a^2b^2 - 4b^2x^2 - 4x^2b^2 = 0 $$

$$ 4a^2b^2 - 8x^2b^2 = 0 $$

$$ 8x^2b^2 = 4a^2b^2 $$

$$ x^2 = \frac{a^2}{2} $$

$$ x = \frac{a}{\sqrt{2}} , x>0 $$

The mistake is in third step while differentiating.

differentiating $\sqrt x$ will give you $\frac{1}{2\sqrt x}$