How is the relation "the smallest element is the same" reflexive?

Why are you testing reflexivity by looking at two different elements of $\mathcal{X}$? The definition of reflexivity says that a relation is reflexive iff each element of $\mathcal X$ is in relation with itself.

To check whether $\mathcal R$ is reflexive, just take one element of $\mathcal X$, let's call it $x$. Then check whether $x$ is in relation with $x$. Because $x=x$, the smallest element of $x$ is equal to the smallest element of $x$. Thus, by definition of $\mathcal R$, $x$ is in relation with $x$. Now, prove that this is true for all $x \in \mathcal X$. Of course, this is true because $\min(x) = \min(x)$ is always true, which is intuitive. In other words, $x \mathcal{R} x$ for all $x \in \mathcal X$, which is exactly what you needed to prove that $\mathcal R$ is reflexive.

You must understand that the definition of reflexivity says nothing about whether different elements (say $x,y$, $x\neq y$) can be in the relation $\mathcal R$. The fact that $\{1\}\mathcal R \{1,2\}$ does not contradict the fact that $\{1,2\}\mathcal R \{1,2\}$ as well.

A binary relation $R$ over a set $\mathcal{X}$ is reflexive if every element of $\mathcal{X}$ is related to itself. The more formal definition has already been given by you, i.e. $$\mathcal{R} \text{ is reflexive iff } \forall x \in \mathcal{X}, x \mathcal{R} x$$

Note here that you've picked two different elements of the set to make your comparison when you should be comparing an element with itself. Also make sure you understand that an element may be related to other elements as well, reflexivity does not forbid that. It just says that every element must be related to itself.