The maximum value of $PA\cdot PB\cdot PC$

The space of quadruples $(A,B,C,P)$ which satisfy constrains is a subset of $(\mathbb R^2)^4$, which I am pretty sure is compact (if we allow degenerate triangles). Since a map which takes a quadruple to $PA\cdot PB\cdot PC$ is continuous, this means that the maximum is achieved somewhere.

Let $(A,B,C,P)$ be a quadruple which achieves this maximum. First, I claim $P$ is not the center of the circle. This is because $P$ being center necessarily gives the product of lengths to be $1$, but we can do better than that.

Next, I claim $P$ lies on one of the sides. This is trivial if $ABC$ is degenerate. So suppose it isn't and $P$ is strictly inside. Take the unique secant of the circle containing $P$. It divides circle into two arcs. Then at least two of the points $A,B,C$ don't lie in the midpoint of the longer arc (call it $M$ for now). Because $P$ is strictly contained in the triangle, if we move one of them a little towards $M$, we will increase its distance from $P$ and hence the product $PA\cdot PB\cdot PC$ (this is very elementary), and we will keep $P$ inside the triangle (this is a bit of handwaving, but I am certain this is true), which is a contradiction with configuration reaching maximum.

So $P$ lies on some side. WLOG, say that $P$ lies on side $AB$. Since $C$ must be at the largest possible distance from $P$, it must lie on the intersection of ray $PO$ with the circle ($O$ is the center). Let $C'$ be intersection of the ray $OP$ with the circle. By power of the point, $PA\cdot PB=PC\cdot PC'$, hence $PA\cdot PB\cdot PC=PC^2\cdot PC'=PC^2(CC'-PC)=PC^2(2-PC)$, as $CC'$ is a diameter. Now we maximize this using calculus or AM-GM inequality, whichever you wish, and we get that the maximum is reached for $PC=\frac{4}{3}$, $PA\cdot PB\cdot PC=\frac{32}{27}$.

It's easy to see from this that the bound is achievable, even by a non-degenerate triangle.


Note: This is a partial answer, since the argumentation is not completely rigorous.

The following reasoning indicates the solution is $\frac{32}{27}$.

Approach: We describe the situation using a coordinate system. We consider the unit circle with center $(0,0)$ and radius $1$.

  • Without any loss of generality we may assume that the position of $C$ is $(1,0)$ and the $y$-coordinate of $P$ is $0$ (by proper rotation of the $x,y$-axes).

  • We take the arguments of @achillehui stated in the comment section and assume that $P$ is located on the side $AB$ of the triangle.

  • Assumption (with less rigor): The point $P$ is located in the middle of the side $AB$. Reasoning: Often extremal problems with ellipses lead to a circle, or extremal problems with rectangles lead to squares.Since we want to find a maximum, we inherently look rather for a symmetrical solution.

Let $\overline{PX}$ denote the distance from $P$ to $X$. Since the product $\overline{PA}\cdot\overline{PB}\cdot\overline{PC}$ should be a maximum, we may assume that the side $AB$ of the triangle is left to the origin, which implies that the $x$-coordinate of $P$ is less than zero.

We are ready for the calculation.

Calculation:

Let $\varphi$ denote the angle $\angle(A0C)$. The distance $\overline{PC}=\overline{P0}+\overline{0C}=\cos(\varphi)+1$ and the distance $\overline{PA}=\overline{PB}=\sin(\varphi)$. We have to determine $\varphi$, so that \begin{align*} f(\varphi)&=\overline{PA}\cdot\overline{PB}\cdot\overline{PC}\\ &=\sin^2(\varphi)(\cos(\varphi)+1)\\ &=1+\cos(\varphi)-\cos^2(\varphi)-\cos^3(\varphi)\quad\rightarrow \quad\max \end{align*}

Setting \begin{align*} f^\prime(\varphi)=\sin(\varphi)(3\cos^2(\varphi)+2\cos(\varphi)-1)=0 \end{align*} we obtain the solutions \begin{align*} \varphi=\begin{cases} n\pi\qquad& n\in \mathbb{Z}\\ \cos^{-1}\left(\frac{1}{3}\right)+2\pi n\qquad& n\in \mathbb{Z}\\ \end{cases} \end{align*}

The relevant solution is $\varphi=\cos^{-1}\left(\frac{1}{3}\right)$ and we finally obtain

\begin{align*} f(\varphi)=f\left(\cos^{-1}\left(\frac{1}{3}\right)\right)=1+\frac{1}{3}-\frac{1}{9}-\frac{1}{27}=\frac{32}{27} \end{align*}