Why don't fractals have more differentiable symmetries?

I do not have a detailed answer, but there is a substantial literature on smooth rigidity of Cantor sets (and other dynamically defined fractals), starting with

D. Sullivan, Differentiable structure on fractal-like sets determined by intrinsic scaling functions on dual Cantor sets. Nonlinear evolution and chaotic phenomena (Noto, 1987), 101–110, NATO Adv. Sci. Inst. Ser. B Phys., 176, Plenum, New York, 1988.

and

D. Sullivan, Differentiable structures on fractal-like sets, determined by intrinsic scaling functions on dual Cantor sets. The mathematical heritage of Hermann Weyl (Durham, NC, 1987), 15–23, Proc. Sympos. Pure Math., 48, Amer. Math. Soc., Providence, RI, 1988.

See for instance:

R. Bamón, C. Moreira, S. Plaza, J. Vera,
Differentiable structures of central Cantor sets.
Ergodic Theory Dynam. Systems 17 (1997), no. 5, 1027–1042.

These papers mainly deal with smooth maps between different Cantor sets, but you should be able to use their results in the setting of a single Cantor set $C$ where you have the extra requirement that $f: C\to C$ sends $x$ to $y$, where $x, y$ are given points.

If you go to www.ams.org/mathscinet and look for papers which refer to the two papers by Sullivan's listed above, you will find many more references.