Is the circle in the square best at avoiding random lines?

Calculus of variation shows that the segment of the boundary of the optimal shape where the tangent line intersects two opposite sides must be a parabola.

To see, this observe that the tangent line is between $(0, y - x \dot{y})$ and $(1, y+ (1-x) \dot{y} )$ so we are trying to maximize the area under the curve $y - x \dot{y}, y+ (1-x) \dot{y} $, which is given by

$$ \int (y - x\dot{y} ) \frac{d}{dx} \left( y+ (1-x) \dot{y} \right) dx $$

$$ = \int (y - x\dot{y} ) (\dot{y} + (1-x) \ddot{y} - \dot{y} ) dx $$

so we can set $$F = (1-x) \ddot{y} (y - x \dot{y} ) - \lambda y$$ and ask to optimize $\int F dx$, which by calculus of variations implies the differential equation $$\frac{dF}{dy} - \frac{d}{d x} \left( \frac{dF}{ d \dot{y}} - \frac{d}{dy} \frac{d^2 F}{d\ddot{y}} \right)= 0$$

We calculate:

$$\frac{d^2 F}{d\ddot{y}} = (1-x) (y - x\dot{y})$$

$$ \frac{d}{dy} \frac{d^2 F}{d\ddot{y}} = - (y - x \dot{y}) + (1-x) ( \dot{y} - \dot{y} - x \ddot{y}) = - (y - x\dot{y} )- x(1-x) \ddot{y} $$

$$ \frac{dF}{ d \dot{y}} =- x (1-x) \ddot{y} $$

$$\frac{dF}{ d \dot{y}} - \frac{d}{dy} \frac{d^2 F}{d\ddot{y}} = y - x\dot{y} $$

$$ \frac{d}{d x} \left( \frac{dF}{ d \dot{y}} - \frac{d}{dy} \frac{d^2 F}{d\ddot{y}} \right) = \dot{y} - x \ddot{y} - \dot{y} = - x \ddot{y} $$

$$\frac{dF}{dy} = (1-x)\ddot{y} - \lambda $$

$$\frac{dF}{dy} - \frac{d}{d x} \left( \frac{dF}{ d \dot{y}} - \frac{d}{dy} \frac{d^2 F}{d\ddot{y}} \right) = \ddot{y} - \lambda $$

and so the solution is a parabola.

In the case of fedja's suggested problem of the maximum area for two opposite lines, we can find the maximum solution with a single boundary segment - it must be given by some equation of the form $y = 1/2 + \alpha (x^2-x +1/6)$ to have area $1/2$, and then the tangent line is between $(0, 1/2 + \alpha ( 1/6 - x^2))$ and $(1, 1/2 + \alpha ( 1/6 - (1-x)^2 ) )$ so the area is given by a $ 1/2 + \alpha/6$ times $1/2 + \alpha/6$ square, minus the corner region given parameterically by $x \to (\alpha x^2, \alpha (1-x)^2)$ which has area $\alpha^2 \int_0^1 (1-x)^2 dx^2 = \alpha^2 \int_0^1 2x (1-x)^2 dx = - \alpha^2/6$ so this is given by optimizing $(1/2 +\alpha/6)^2 - \alpha^2/6$, which is achieved when $2/6 (1/2+\alpha/6) = 2 \alpha/6$, or $\alpha=3/5$, and the probability achieves is $1/2 \left( ( 1/2 + \alpha/6)^2 - \alpha^2/6 \right) = (1/2) (3/5)^2 (5/6) = 3/20$

So the parabolic region $y \geq 1/2 + (3/5) (x^2-x+ 1/6)$ achieves a probability of $3/20$ of a random line through two opposite sides intersecting it, which beats the $1/8$ achieved by the region $y \geq 1/2$ and, I suspect, is optimal for that subproblem.

I guess it won't hurt to post the details of the solutions to the "toy" problems: the separate cases of lines through the adjacent sides and through the opposite sides. Let us consider the adjacent sides case first. It is really cute (IMHO) and I wish that were what was requested. Alas, the life is never easy :-)

I will parameterize the convex domain by the family of its tangent lines. As the point $p$ runs over the boundary of our convex domain (which I may assume to be reasonably smooth and located strictly inside the square without loss of generality) counterclockwise, the endpoints $x$ and $y$ of the corresponding tangent line run counterclockwise along the boundary of the square $x$ being ahead of $y$ (see the picture).

The corresponding infinitesimal butterflies between "successive" tangent lines sweep the area between our shape and the surrounding square twice. Identifying the points $x,y$ with their distances to the "common corner" (the orange case) or the "common side" (the red case), we see that the butterfly area elements are $$ \frac 12 \frac{(\frac Xx)^2+(\frac Yy)^2}{\frac Xx+\frac Yy}xy\text{ and } \frac 12\frac{X^2+Y^2}{X+Y} $$ respectively where $X=|\dot x|$ and $Y=|\dot y|$ are the speeds of $x$ and $y$. These expressions should integrate to $1$ (twice $\frac 12$). On the other hand, the integral expression for the probability in question involves $$ \frac 18(Xy+Yx)=\frac 18\left(\frac Xx+\frac Yy\right)xy $$ in the corner case and $\frac 18(X+Y)$ in the side case (recall that we do not allow the lines through parallel sides here, so only the right side of the square can be used as the location of the second point for $x$ or $y$ in the red configuration). By Cauchy-Schwarz, the probability element is at most half of the butterfly area element in all cases, whence we can bound the probability by the area between the shape and the surrounding square. In order to have equality, we want the regime with $\frac Xx=\frac Yy$ in the corner case and $X=Y$ in the side case. The first one results in $xy=\operatorname{const}$, i.e., in an arc of hyperbola. The second one corresponds to the rotation around a point. So, we get a shape consisting of 4 hyperbolic arcs. It remains to choose the constant to get the area as requested.

to be continued eventually but not necessarily soon

Indeed it seems that the circle is not optimal although the improvements I've found only raise the probability slightly and are visually hard to distinguish from a circle. Here is the circle in red and a region $\Delta_h$ bounded by $4$ hyperbolas. Recall that the circle has $p(\Delta_0)=0.34470989\dots.$ It turns out that $p(\Delta_h)=0.34494\dots.$

It was suggested that the optimum might be a shape bounded by $4$ hyperbolas near the corners and $4$ parabolas near the middles of the sides. The best I could do with that is $\Delta_{hp}$ shown here with $p(\Delta_{hp})=.34544686\dots.$

The parabola at the bottom is roughly $y=1.221614(x-\frac12)^2+.1045232.$ The transition from the parabola to the hyperbola to its left occurs between $(0.26957, 0.16938)$ and $(0.267287, .170719)$ (the respective endpoints of the polygonal approximations) with the middle of the hyperbola at $(0.213614,0.213614).$

Some disclaimers: I am not certain this is correct. I expected that the contact point of a tangent line rolling around would transition from hyperbola to parabola exactly when its intersection of the line with the square passes a corner and begins to intersect opposite sides. I believe this would maintain a constant area between the tangent and the square.However I don't see that happening here. Also, The later digits may not be exact, I tried to truncate before a small calculated digit. These are actually polygons obtained by sampling (at equally radially spaced points). The area may come out to be less than $\frac12$ in which case the figure is dilated appropriately. This will map parabolas to parabolas but deform hyperbolas. However replacing the deformed sections by a best hyperbola approximation changes the area only slightly. Iterating a few times stabilizes. I did not do this adjusting to $\Delta_h$ so it may be cheated out of a bit.

I don't know if this is really the optimal shape. I could investigate small perturbations of the polygon but have not. I am pretty sure that something similar to a quarter circle in a corner OR a parabola along with most of the top OR the top, with the upper portions of the two sides connected by the arc of a conic could not do as well. I'm not saying it would be hard to rule those out. I just haven't rigorously done so.