How is the free energy of Kosterlitz-Thouless transition smooth yet non-analytic?

First, the expression given in the OP is not the expression for the actual free energy, only what comes out of the naive heuristic energy/entropy argument.

In reality, renormalization group computations lead to the following predictions: first, the correlation length should blow up at the transition as $$ \xi \simeq A\exp\bigl( B/\sqrt{t} \bigr) $$ for $t>0$ ($\xi$ is infinite for $t\leq 0$), where $t=(T-T_{\rm BKT})/T_{\rm BKT}$ is the reduced temperature. Observe how this is dramatically faster than the more common power-law divergence of the correlation length at a usual critical point.

Second, the singular part of the free energy should satisfy $f_{\rm sing} \sim \xi^{-2}$, that is, $$ f_{\rm sing} \simeq C \exp\bigl( -2B/\sqrt{t} \bigr) $$ for $t>0$ small.

Note that the function $$ t\mapsto \begin{cases} \exp\bigl( -2B/\sqrt{t} \bigr) & \text{for }t>0\\ 0 & \text{for }t\leq 0 \end{cases} $$ is infinitely differentiable but not analytic at $t=0$, since one does not recover the original function by summing its Taylor series. This is what is meant by "smooth yet not analytic" in this context.

I am not a specialist, so I won't go into more detail here. There are no mathematically rigorous proofs of the above claims in the XY model (even the proof of the existence of the Kosterlitz-Thouless phase transition requires rather sophisticated mathematical arguments). There are, however, other simpler examples of phase transitions in which this type of "smooth but non analytic behavior" are found and for which rigorous results are available.

If you want to read more about these issues in the XY model, you can look at Kosterlitz's original paper (see also his recent review). You can also read about that in several textbooks, for instance this one (Itzykson and Drouffe) and this one (Kardar).