What length would the sides of a triangle over Earth's surface be for the sum of its angles to be 180.1°

The area of a spherical triangle is exactly $ER^2$ where $E$ is the angular excess. In our case $E=\frac{\pi}{1800}$, so the given triangle covers $\frac{1}{7200}$ of Earth's surface. Assuming $R=1$, l'Huilier's formula (page 184 of my notes) relates the angular excess / the area to the semiperimeter / the side lengths through $$ \tan\frac{E}{4}=\sqrt{\tan\frac{s}{2}\tan\frac{s-a}{2}\tan\frac{s-b}{2}\tan\frac{s-c}{2}} $$ and in our case we have $a=b=c=\ell$ and $s=\frac{3}{2}\ell$, so

$$ E = \frac{\pi}{1800} = 4\arctan\sqrt{\tan\frac{3\ell}{4}\tan^3\frac{\ell}{4}} $$ and by solving $$ \tan\frac{3\ell}{4}\tan^3\frac{\ell}{4}=\tan^2\frac{\pi}{7200} $$ we get that $\ell$ is approximately $6.347\%$ of the radius $R$.

For Earth, $R =6\ 371$ km, and $\ell = 404.377$ km.

Let $r$ be the radius of the Earth; the area of a triangle between the equator and two meridians $90^\circ$ apart is $\frac{r^2\pi}{2}$ and has a defect of $270^\circ-180^\circ=90^\circ$. Now we want a triangle with $900$ times smaller defect, which means it needs to have $900$ times smaller area: $\frac{r^2\pi}{1800}$.

Now I will not calculate the side for the spherical equilateral triangle (for the given area) but a planar one, expecting (without proof!) that this won't make a lot of difference. Thus, the side is approximately $\sqrt{\frac{4}{\sqrt 3}\frac{r^2\pi}{1800}}=r\sqrt{\frac{\pi}{450\sqrt 3}}\approx 404\text{ km}$.

Consider an equilateral spherical triangle with side lengths $a$ and angles $A$, by the supplemental cosine rule https://en.wikipedia.org/wiki/Spherical_trigonometry#Identities

\begin{eqnarray*} \cos(A) =- \cos^2(A)+\sin^2(A) \cos(a) \\ \cos(a) =\frac{\cos(A)+\cos^2(A)}{\sin^2(A)} =\frac{\cos(A)}{1-\cos(A)} \end{eqnarray*} Now we have $A=60.0333\cdots$, which gives $a=3.6\cdots$.

To obtain the distance on a sphere of radius $r$ calculate $ra\pi/180$.