Is there a smooth manifold which admits only rigid metrics?
The answer to the question in the first sentence is "yes". Let $M$ be a hyperbolic 3-manifold whose isometry group is trivial. Then by Theorem 1.1 of
Farb, Benson; Weinberger, Shmuel Hidden symmetries and arithmetic manifolds. Geometry, spectral theory, groups, and dynamics, 111–119, Contemp. Math., 387, Amer. Math. Soc., Providence, RI, 2005. (Reviewer: Bachir Bekka) 53C35 (53C23)
(which they attribute to Borel, though they do not give an original reference for it), the isometry group of every Riemannian metric on $M$ is isomorphic to a subgroup of the hyperbolic isometry group, and thus is trivial.
Along similar lines, you might be interested in Theorem H of
Dinkelbach, Jonathan and Leeb, Bernhard, Equivariant Ricci flow with surgery and applications to finite group actions on geometric 3-manifolds. Geom. Topol. 13 (2009), no. 2, 1129–1173.
It says that every finite-order diffeomorphism of a closed hyperbolic 3-manifold is smoothly conjugate to an isometry, so closed hyperbolic 3-manifolds with trivial isometry groups give examples of smooth manifolds with torsion-free diffeomorphism groups.
The survey paper "Do manifolds have little symmetry?" by Volker Puppe lists several known results about manifolds with no nontrivial finite group action. The ArXiv link is http://arxiv.org/pdf/math/0606714v1.pdf
In particular
- aspherical manifolds with $Z(\pi_1M)=0$ and $Out(\pi_1M)$ torsionfree
have no finite group action (Borel)
Examples are
- certain mapping tori of nilmanifolds (Conner-Raymond-Weinberger)
- certain hypertoral manifolds, i.e., n-manifolds with a degree 1 map to the n-torus (Schultz)
- certain 3-manifolds (Edwards)
- a certain Bieberbach manifold (Waldmüller)