Infinite Sequence of Inscribed Pentagrams - Where does it converge?

Your intuition about a projective transform was correct: a projective transformation relates one iteration to the next, they don't even skip a step. A quick experiment with Cinderella verified this pretty easily. By now I even have a proof, so this answer differs greatly from previous versions.

Mapping corners for one step

This section is a proof that the transformation mapping the outer pentagram to the inner one is in fact a projective transformation. If you prefer to simply take my word for it, feel free to skip to the next section.

A projective transformation is uniquely defined by the images of four points, no three of which may be collinear. So suppose you define the projective map $T$ using the following four points:

\begin{align*} A_1&\mapsto A_2& B_1&\mapsto B_2& C_1&\mapsto C_2& D_1&\mapsto D_2 \end{align*}

Now you know that $T$ preserves incidences, so (using $\vee$ to denote “join” and $\wedge$ to denote “meet”) you get

\begin{align*} T(E_2) &= T\bigl((A_1\vee B_1)\wedge(C_1\vee D_1)\bigr) \\&=\bigl(T(A_1)\vee T(B_1)\bigr)\wedge(T(C_1)\vee T(D_1)\bigr) \\&=(A_2\vee B_2)\wedge(C_2\vee D_2) = E_3 \end{align*}

You may also know that a projective transformation will preserve cross ratios. Therefore you get $(A_1,B_1;E_2,C_2)=(A_2,B_2;E_3,T(C_2))$. So once you know $A_2$, $B_2$ and $E_3$ (as we do), the point $T(C_2)$ with relations to these has to be fixed. You can construct this point, using the fact that a perspectivity will preserve cross ratios. A perspectivity centered at $D_2$ will map

\begin{align*} A_1&\mapsto B_2& E_2&\mapsto C_3& C_2&\mapsto E_3& B_1&\mapsto A_2 \end{align*}

This is a reversal of points, but reversing the order of points in the cross ratio will not change its value. So by now you have fixed $E_3$ by intersecting two lines, and now have shown $T(C_2)$ to be the intersection of $A_2\vee B_2$ with $D_2\vee E_2$. Therefore you now know that $T(C_2)=C_3$. With a similar argument you get $T(B_2)=B_3$. And now you can use these to construct $T(E_1)$:

\begin{align*} T(E_1) &= T\bigl((A_1\vee B_2)\wedge(D_1\vee C_2)\bigr) \\&=\bigl(T(A_1)\vee T(B_2)\bigr)\wedge(T(D_1)\vee T(C_2)\bigr) \\&=(A_2\vee B_3)\wedge(D_2\vee C_3) = E_2 \end{align*}

So at this point you know that the fifth point of the outer pentagram will be mapped to the fifth corner of the inner pentagon, even though only the other four points define that map.

Iterative transformations

So what is the transformation which maps the second to the third pentagram? It is the same: as that transformation preserves incidence, it maps the first pentagram and its inscribed pentagram to the second pentagram and its inscribed pentagram, i.e. the third pentagram. So you are in fact asking about the limit of repeated applications of a single transformation.

Limit case

Now consider the limit case. Your projective transformation $T$ is a $3\times 3$ matrix, and its eigenvectors correspond to the fixed points of the transformation. Consider the diagonal form of that matrix. Repeating the transformation means taking powers of that diagonal matrix. But as we are dealing with homogenous coordinates, you can rescale that matrix arbitrarily. This means that in the long run, only the eigenvalue with the maximal absolute value will have any contribution, the other two eigenvalues will converge to zero after rescaling.

So the homogenous coordinates of the convergence point are given by the eigenvector corresponding to the maximal (absolute) eigenvalue.