Pythagorean theorem expressed without roots in an old Tamilian (Indian) statement

Let the legs measure $a$ and $ka$ with $k>1$. Then the approximation given is:

$\frac{7ka}{8}+\frac{4a}{8}=\frac{(7k+4)a}{8}$.

The real measure is $\sqrt{k^2a^2+a^2}=\sqrt{k^2+1}a$.

So how does $\frac{7k+4}{8\sqrt{k^2+1}}$ behave? Not that bad.

Here is a graph:

So as the graph shows, in the interval $(1,\infty)$ the best approximation occurs at $k=1$ for which we get $\frac{7+4}{8\sqrt{2}}=\frac{11}{2\sqrt2}\approx 0.972$. After this the ratio becomes smaller and smaller, the limit is $\frac{7}{8}=0.875$.

Can this be improved? yes it can, the formula always gives us a length shorter than the actual length, so we can obtain a better aproximation by taking slightly larger coefficients.

A better question is which is the best aproximation for $\sqrt{a^2+b^2}$ that is of the form $la+mb+n$ with $l,m,n\in \mathbb Q$. Now, the value of $l$ is going to be irrelevant because when $a$ and $b$ are large enough the $l$ won't matter much.

So we need to approximate $\sqrt{a^2+b^2}$ with $la+mb$. If we write $b$ as $ka$ then we need to approximate $\sqrt{k^2+1}a$ with $l+mk(a)$. So essentially what we need to do is approximate $\sqrt{k^2+1}$ with $l+mk$. This is the real problem.

The approximation for $\sqrt{k^2+1}$ provided in the question is $\frac{7k+4}{8}$. Now, if we were to approximate $\sqrt{k^2+1}$ by $mk+l$ I would think it would be in our best interest to make $m=1$ (so that at least when $k$ goes to infinity the limit becomes $1$. Then it is only a matter of finding a good $l$.

I didn't think a lot but taking $l=\frac{3}{7}$ seems to give a good result. Here is the graph of $\frac{k+3/7}{\sqrt{k^2+1}}$

This is a better approximation, the ratio of the correct measurement, versus the actual measurement when $k\geq 1$ reaches a maximum of approximately $1.088$ when $k=2.\overline 3$, this is the worst it gets, it improves when $k$ approaches $1$, reaching the minimum of $1.01$ and it also improves as $k$ goes to infinity, with a limit of $1$. (as an opposite of the first approximation this approximation always gives a hypotenuse longer than it actually is, which again tells us this is not actually the best approximation)

So a better way to approximate the hypotenuse is to add the longer leg's length plus three sevenths of the shorter leg's length.

The method provides a not-so-great approximation when the two catheti are roughly of the same length. In this case, $$\frac{7}{8}a+\frac{1}{2}b\approx\frac{7}{8}a+\frac{1}{2}a=\frac{11}{8}a=1\mathrm.375 a\approx(1\mathrm.414...)a=\sqrt{2}{a}\approx\sqrt{a^2+b^2}.$$

Numerical optimization reveals how truly astonishing the approximation is:

Taking a hint from @dREaM's answer and normalizing the length of the shortest side, for side lengths $1$ and $k$ we are interested in the ratio

$$\frac{ak+b}{\sqrt{k^2+1}}$$

as a function of the coefficients $a$ and $b$ (in the Tamil approximation, $a=7/8$ and $b=1/2$). We want the ratio to be as close to $1$ as possible for a large number of side lengths. Some experimentation shows that it is the region $k \in [1,3]$ that is the most problematic, so it makes sense to minimize the integral

$$ \int_1^3 \left(1- \frac{ak+b}{\sqrt{k^2+1}}\right)^2 \>dk $$

for $a,b$. Numerically, the minimum lies near

$$ \begin{align} a &= 0.871079 \\ b &= 0.509221 \end{align} $$

which is very close to the values given in the original text. In fact, plotting the integrand for the original (orange) versus the numerically obtained (blue) coefficients shows that the values are very near optimal in the range $[1,3]$, and most of the discrepancy happens where the side lengths are close to each other (that is, where $k$ is close to $1$):

If we assume that most practical right triangles have short sides that are within a factor of $3$ of each other, then given the very simple fractions used there can be no doubt that the original approximation is "optimal" in a practical sense.