Maximum area of triangle inside a convex polygon

There is not-so-painful way to prove a weaker inequality, i.e. that for any convex polygon $P$ with unit area there is a triangle $T\subset P$ with area $\geq \color{purple}{\frac{4}{21}}$. The approach is simple: let, counter-clockwise, $P_1,P_2,\ldots,$ $P_n=P_0,$ $ P_{n+1}=P_1$ be the vertices of the polygon, and let $P_{a-1}P_{a} P_{a+1}$ be the triangle with the smallest area among all the triangles of such a form. We remove the vertex $P_a$ and re-index the vertices, then continue the removal procedure until our polygon is a triangle. That is our $T$. It is not difficult to check that if the area of $P$ is $1$, the area of $T$ is at least $$ \prod_{n\geq 4}\left(1-\frac{4\sin^2\frac{\pi}{n}}{n}\right)\geq \frac{4}{21}. $$ Some comments of mine to your revised question may be useful to improve the above inequality/approach: for instance, there is for sure a triangle $T\subset P$ with area $$ \frac{3\sqrt{3}}{4\pi} \Delta(E^-) $$ where $E^-$ is the John-Loewner inellipse (the inscribed ellipse in $P$ with maximal area), and we have very accurate bounds when the vertices of $P$ all lie on a ellipse (so for any $n\leq 5$).

The original Blaschke argument is outlined in this answer by Christian Blatter.
Blaschke's approach is extremely slick: there is a Steiner symmetrization procedure that is area-preserving, so we may assume without loss of generality that the original polygon is a cyclic polygon. But that gives a way easier optimization problem, just related with the distribution of $n$ points on a circle.

As first pointed out by Mark Fischler in his comment, one can take $c = \frac{3\sqrt{3}}{4\pi}$.

In addition to Steiner symmetrization mentioned in Jack D'Aurizio's answer$\color{blue}{{}^{[1],[2]}}$, there is another elegant analytic approach which can be generalized to inscribed $n$-gon.

E. Sas (1939) - For any $n \ge 3$, let $c_n = \frac{n}{2\pi}\sin\left(\frac{2\pi}{n}\right)$. For any convex body $B$ in the plane, there exists a $n$-gon $P$ inside $B$ such that $$\verb/Area/(P) \ge c_n \verb/Area/(B) \tag{*1}$$

When $n = 3$ and $B$ is a convex polygon, the claim we can take $c = c_3 = \frac{3\sqrt{3}}{4\pi}$ follows immediately.

Following argument is based on a paper by E. Sas in German$\color{blue}{{}^{[3]}}$. All mistakes are mine and in case anything looks fishy. Please refer to E. Sas paper for correct statement.

---

Since I am lazy, I will assume $B$ is a convex body whose boundary $\partial B$ is a smooth Jordan curve. This avoids all sort of potential pathologies and save me from justifying a lot of stuff.

Let $2\ell$ be diameter of $B$. Let $L$, $R$ be two points on $\partial B$ at a distance $2\ell$ apart. Choose a coordinate system such that $L,R$ is located at $(-\ell,0)$ and $(\ell,0)$ respectively. Under such a coordinate system, $\partial B$ has a parametrization $\gamma$ of the form:

$$[0,2\pi] \ni t \quad\mapsto\quad \gamma(t) = (\ell\cos t, e(t)\sin t ) \in \partial B$$

where $e(t) > 0$ is some smooth function. Extend $\gamma(t)$ and $e(t)$ to smooth periodic functions over $\mathbb{R}$ with period $2\pi$.

For any fixed $t \in [0,2\pi]$ and $k \in \mathbb{Z}$, let $t_k = t + \frac{2k\pi}{n}$. It is clear $t_{k+n} = t_k$.

Let $P(t)$ be the $n$-gon with vertices $\gamma(t_0), \gamma(t_1), \gamma(t_2), \ldots, \gamma(t_{n-1})$. Since $B$ is convex and $\gamma(t_k) \in B$, $P$ lies inside $B$.

Let $f(t)$ be the area of $P(t)$. It is easy to work out

$$f(t) = \frac{\ell}{2} \sum_{k=1}^n e(t_k)\sin t_k(\cos(t_{k-1}) - \cos(t_{k+1})) = \ell\sin\left(\frac{2\pi}{n}\right)\sum_{k=1}^n e(t_k)\sin^2(t_k)$$

Now treat $t$ as a variable and average $f(t)$ over $[0,2\pi]$, one get

$$\frac{1}{2\pi}\int_0^{2\pi} f(t) dt = \frac{n}{2\pi}\sin\left(\frac{2\pi}{n}\right)\times \ell\int_0^{2\pi} e(t)\sin^2(t)dt = c_n \verb/Area/(B)$$

This implies there exists a $t_{*} \in [0,2\pi]$ such that $f(t_{*}) \ge c_n \verb/Area/(B)$. In other words, there exists a poylgon $P = P(t_{*}) \subset B$ whose area is at least $c_n$ of that of $B$.

Back to the problem at hand for convex polygon.

For any polygon $P$, let $|P|$ be its number of sides. Given any convex polygon $Q$ and any $0 < c < c_n$, approximate $Q$ by a convex body $B$ with smooth boundary whose area is at least $\frac{c}{c_n}\verb/Area/(Q)$. By $(*1)$, there is a $n$-gon $P \subset B \subset Q$ with $\verb/Area/(P) \ge c_n\verb/Area/(B) \ge c\verb/Area/(Q)$. Since the set of polygon $P \subset Q$ with $|P| \le n$ is compact under the topology induced from $\mathbb{R}^2$ and $\verb/Area/(\cdot)$ is continuous with respect to this topology, there is a polygon $P_{*} \subset Q$ with $|P_{*}| \le n$ and $\verb/Area/(P_*) \ge c_n \verb/Area/(Q)$. If $|P_{*}| < n$, we can turn $P_*$ to a $n$-gon by adding some extra vertices along its edges.

Fix $n$ to $3$, this means there is a triangle inside $Q$ whose area at least $c_3 = \frac{3\sqrt{3}}{4\pi}$ of that of $Q$.

Notes

• $\color{blue}{[1]}$ - For a proof of $(*1)$ when $n = 3$, see this answer by Christian Blatter.

• $\color{blue}{[2]}$ - Christian Blatter's answer uses Steiner symmetrization. For more detail, please refer to
  Wilhelm Blaschke, Über affine Geometrie III: Eine Minimumeigenschaft der Ellipse. Leipziger Berichte 69 (1917), pages 3–12.

• $\color{blue}{[3]}$ - E. Sas, Über eine Extremaleigenschaft der Ellipsen, Compositio Math. 6 (1939), 468–470.

The statement is true using any $c < \frac{3\sqrt{3}}{4\pi} \approx 0.413$, which is more than $ \frac38$. But I have tried and failed to find an easy proof.

EDITED A DAY LATER

If you impose a restriction on the number $s$ of sides in the convex polygon, the $c$ for that set of polygons becomes greater. For $s=3$, $c=1$ of course. For $s=4$ it is easy to show that $c=\frac12$: $c$ is at least $\frac12$, as demonstrated by a line connecting two non-adjacent vertices (one of the triangles thus formed must have at least half the area); and $c$ is at least $\frac12$, as demonstrated by a square.