Limit of a sequence of polygons.

Corrected per Thorny's comment: The limit curve does not appear smooth, as is visible in this picture of the first 12 polygons obtained by the process, which become visually indistiguishable at the end:

alt text http://dl.dropbox.com/u/5390048/chippedsquare.jpg

Despite appearances, and despite my initial assertion, the limit curve actually is smooth. Here is a sequence of closeups of one of the apparent corners, enlarging by a factor of 2 in successive images:

alt text http://dl.dropbox.com/u/5390048/CornerCloseup.jpg

As Thorny pointed out, for the midpoint of any edge, there is an affine transformation fixing the point and taking the limit curve to itself, shrinking the edge by a factor of 3, and shrinking the plane mod the tangent to the edge by a factor of 3, and with only one eigenvector, which is tangent to the edge. Therefore, despite my initial impression, the limit curve is actually tangent to the $k$th stage polygon at the midpont of each of its edges. It looks sharp because the slopes converge slowly.

To understand this better, think of a new process scaled up by a factor of 3. If we describe the polygon by a sequence of vectors $\{v_1, \dots, v_k\}$ representing the sides, at each stage we interpolate the sum of each pair of successive vectors in circular order. The slopes of the vectors are rational, and the interpolation is by Farey addition (adding numerators and denominators). For instance, if we start with the vectors $(1,0)$ and $(0,1)$, then after 4 subdivisions we get this sequence of slopes:

{0, 1/5, 1/4, 2/7, 1/3, 3/8, 2/5, 3/7, 1/2, 4/7, 3/5, 5/8, 2/3, 5/7, 3/4, 4/5, 1, 4/5, 3/4, 5/7, 2/3, 5/8, 3/5, 4/7, 1/2, 3/7, 2/5, 3/8, 1/3, 2/7, 1/4, 1/5}

All rational slopes are obtained in this way. Since the curves are convex and the slopes are dense in the limit, the limiting curve is $C^1$, but it is not twice differentiable everywhere.

You can't expect this kind of curve to have a name (in most cases).

You've described de Rham's trisection method (Un peu de mathématiques à propos d'une courbe plane. Elemente der Math. 2, (1947). 73–76, 89–97.) It's the first example of a corner-cutting method. For an analysis of convergence, see Carl de Boor's Cutting corners always works (Comput. Aided Geom. Design 4 (1987), no. 1-2, 125–131). See also Chaikin's corner-cutting method and subdivision surfaces.

The left figure shows the first three iterations, the right after 10 iterations, when the polygon has 4096 vertices.