An open set invariant under a linear map implies it is an isometry or of finite order?

You don't need to assume that $0\in U$.

Since $AU=U$, all orbits of vectors in $U$ completely lie in $U$. Since $U$ is bounded, no vector in $U$ can have an unbounded orbit (because an unbounded orbit would not lie in $U$). Since $U$ is open, it spans all of $\mathbb R^2$, that is, any vector in $\mathbb R^2$ can be written as linear combination of vectors $u_i$ in $U$. But then for a general vector $v\in\mathbb R^2$, we have $$\|A^kv\| = \|A^k(\sum_i \alpha_i u_i)\| = \|\sum_i \alpha_i (A^k u_i)\| \le \sum_i \|\alpha_i (A^k u_i)\| = \sum_i |\alpha_i| \|A^k u_i\|$$ and therefore the orbit of $v$ is bounded because all orbits of $u_i$ are.

This is false. That is, there is an element $A\in Sl_2$ which preserves a connected bounded open set $U$, but for which $A$ is neither orthogonal, nor finite order. In fact, there are many such $A$.

Let's construct some. To begin with, let $B =\begin{bmatrix} \cos \theta & -\sin \theta \\ \sin\theta & \cos\theta\end{bmatrix}$ be a rotation matrix, where $\theta$ is in irrational multiple of $\pi$. In particular, $B$ has infinite order. In fact, $\langle B\rangle$, the subgroup of $SO(2)$ generated by $B$, is dense.

Now, $SO(2)$ is not normal in $Sl_2$. This implies that there is a matrix $C\in Sl_2$ for which $CBC^{-1}\notin SO(2)$. (For, if $CBC^{-1}\in SO(2)$ for every $C\in Sl_2$, then $C\langle B\rangle C^{-1}\subseteq SO(2)$. Then, since $SO(2)$ is closed $\overline{ C\langle B\rangle C^{-1}}\subseteq SO(2)$. But $\overline{C \langle B\rangle C^{-1}} = C\overline{\langle B\rangle}C^{-1} = C SO(2) C^{-1}$. This shows that $SO(2)$ is normal in $Sl_2$, giving a contradiction.)

Choosing a $C$ with $CBC^{-1}\notin SO(2)$, I claim that $A = CBC^{-1}$ meets all your guidelines. It's not orthogonal by construction, and since $CBC^{-1}$ and $B$ have the same order, it's not of finite order.

So, we need only construct $U$. To that end, note that $B$ preserves the ball of radius $1$ centered at $(0,0)$, because $B$ is orthogonal. Calling this ball $V$, this just means $BV = V$. Let $U = CV$.

Note that $$AU = CBC^{-1} CV = CBV = CV = U,$$ so $U$ is preserved. Since left multiplication by $C$ is an isomorphism, it is, in particular a homeomorphism, so $U$ is connected and open. Lastly, $U$ is bounded because $C$ has bounded operator norm.

Edit

To answer the question in the comments...

Proposition: Suppose $A\in Sl_2$ preserves a bounded open set $U$. Then $A$ is conjugate to an element in $SO(2)$.

Proof: From Celtschk's answer together with the OPs observation, the hypothesis that $A$ preserves an open set $U$ implies that $A$ is diagonalizable. Let's assume first that the eigenvalues $r$ and $\frac{1}{r}$ have different magnitudes. This, in particular, implies they are real numbers. By swapping $r$ and $\frac{1}{r}$, we may thus assume that $|r| > 1$. Let $v_1$ be an eigenvector for $r$ and $v_2$ be an eigenvector for $\frac{1}{r}$. As mentioned in Celtschk's answer, there is a vector $v = a_1 v_1 + a_2 v_2$ with $a_1\neq 0$. Then, by induction, $A^n v = r^n a_1 v_1 + \frac{1}{r}^n a_2 v_2 \in U$ for all $n = 1,2,3,...$. Note that $|A^n v| \geq |r|^n |a_1| \rightarrow \infty$ as $n\rightarrow \infty$. So, we have contradicted the fact that $U$ is bounded. This concludes the case where the eigenvalues of $A$ are real.

So, we may assume that the eigenvalues $|r|$ and $|\frac{1}{r}|$ have the same magnitude, which must therefore be $1$. We will show that the closure of the subgroup generated by $A$, $\overline{\langle A\rangle}$ is compact. Once we've done this, the Cartan-Iwasa-Malcev theorem tells us that $\overline{\langle A\rangle}$ is conjugate to a subgroup of $SO(2)$, so $A$ is conjugate to something in $SO(2)$.

Now, let's show that $\overline{\langle A\rangle}$ is compact. Once and for all, choose a (possibly complex) matrix $C$ which diagonalizes $A$, so $CAC^{-1}$ is diagonal.

For a (possibly complex) matrix $D = (d)_{ij}$, let $ \|D\| = max(|d_{ij}|)$. Note that the inequality $\|AB\| \leq 2\|A\| \|B\|$ holds since any entry of $AB$ is of the form $a_1 b_1 + a_2b_2$ where the $a_i$ are entries in $A$ and the $b_j$ are entries of $B$.

Note also that $\|(CAC^{-1})^n\| = 1$ since $(CAC^{-1})^n$ is diagonal with diagonal entries $\lambda^n$ and $(1/\lambda)^n$.

Thus, we see that $\|A^n\| = \| C^{-1}(CAC^{-1})^n C\| \leq 4\|C^{-1}\| \| (CAC^{-1})^n\| \|C\| = 4\|C^{-1}\| \|C\|$. That is, every entry of $A^n$ is bounded, independent of $n$.

Thus, the cyclic subgroup generated by $A$ is bounded, and so, therefore is its closure. That is $\overline{\langle A\rangle}$ is a closed bounded subset of $Sl_2$. Since $Sl_2$ is closed in $M_2(\mathbb{R})$ (because it's the inverse image of a point under the continuous $\det$ map), $\overline{\langle A\rangle}$ is closed and bounded in $M_2(\mathbb{R})\cong \mathbb{R}^4$. Thus, it is compact. $\square$

I'm probably (still) missing something, but...

We know $A$ is a real 2x2 matrix of determinant 1, and all eigenvalues have modulus 1, from @celtschk's answer. From this, we know that the eigenvalues are either $\exp(it), \exp(-it)$ or $r, 1/r$, where $t$ and $r$ are real. In the first case, $A$ is a rotation by angle $t$, hence orthogonal. In the second case, the fact that the eigenvalues have modulus $1$ means that they are both $+1$ or both $-1$, so $A$ is either the identity or $-I$.

Probably @celtschk figured all this was trivial and didn't both writing it out, but I've done so for clueless folks like me who come upon this question later.

Post-comment addition Let $z = \exp(it) = c + i s$, where $c = \cos t, s = \sin t$, so the other eigenvalue is $\bar{z}$. Let $v_1$ be a (complex) eigenvector for $z$, so $$ Av_1 = zv_1. $$ Conjugating both sides gives us $$ A \bar{v}_1 = \bar{z} \bar{v}_1 $$ (because $A$ is real), so we have an eigenvector $v_2 = \bar{v}_1$ for the other eigenvalue, too.

I believe that if you let $$ w_1 = \Re(v_1); w_2 = \Im(v_1) $$ then you find that $w_1$ and $w_2$ are orthogonal vectors in $\Bbb R^2$, and that $$ Aw_1 = \cos(t) w_1 + \sin(t) w_2 \\ Aw_2 = \sin(t) w_1 - \cos(t) w_2 $$ so that in the $w$ basis, $A$ is a rotation by $t$...but then it's a rotation by $t$ in any basis.

As I said, I believe this, but I can't actually get the algebra right, and I've got a deadline in a couple of hours, so I have to stop; I hope that this gets you going in the right direction (or lets you see why I'm completely wrong!).