What's the difference between "duality" and "symmetry" in mathematics?
No, duality and symmetry are not the same thing. Although in many contexts "the dual of" is a symmetric relation, this is not invariably the case (e.g. the dual of the dual of a topological vector space need not be the original).
Moreover symmetry is not just about symmetric relations; it has to do mainly with automorphisms of algebraic, geometric or combinatorial structures. Those structure preserving automorphisms (including trivial the identity mapping) form a group, and we'd refer to it as the symmetry group of the structure.
As you note, there are many kinds of symmetry. Some symmetries have order two but many do not. Indeed the group of symmetries may combine elements that have finite order with those having infinite order, elements that have discrete action with some that are continuous mappings. The symmetries of a right circular cylinder, for example, would include discrete actions like reflection in a midplane as well as continuous actions of rotation about the axis.
If you are looking for a fundamental difference, perhaps it should be noted that duality often involves different categories, i.e. the dual may belong to a different category than the original, while symmetry involves not only the same category but actually a mapping of the same object to itself.
An algebraic, geometric or combinatorial object $\Omega$, consisting of a "ground set" $O$, and provided with additional structure elements like a metric, edges, binary operations, etc., may have symmetries. A symmetry is the same thing as an automorphism, i.e., a bijective map $\phi: \ O\to O$ such that any relevant relation among the elements $x\in O$ is preserved. This means that if, e.g., it matters that $x*y=z$ then one should have $\phi(x)*\phi(y)=\phi(z)$. Sometimes a symmetry $\phi$ has the property that $\phi$ is not the identity, but $\phi\circ\phi$ is. Such a $\phi$ is called an involution.
Now the concept of duality is a different matter. A duality is an involution not of a single object, but of a whole theory. Examples are the duality present (a) in plane projective geometry or (b) in the theory of convex polyhedra.
Ad (a): To each theorem in PPG corresponds its dual theorem. The dual of Pascal's theorem about $6$ points on a conic is Brianchon's theorem about $6$ tangents to a conic.
Ad (b): An individual polyhedron, say a platonic solid, may have an interesting set of symmetries. But duality is something far deeper. It says that to any such polyhedron $P$, symmetric or not, corresponds a dual polyhedron $\hat P$, such that incidences among the vertices, edges, and faces of $P$ appear in $\hat P$ reversed. The dual of an dodecahedron is an icosahedron.