If $f$ is continuous and $f(f(x))+f(x)+x=0$ for all $x$ in $\mathbb R^2$, then $f$ is linear?

No, such an $f$ need not be linear. Explicitly writing down a counterexample is rather messy (unless I'm missing some trick), but here's the idea. Fix three points $p$, $q$, and $r$ such that $p+q+r=0$, a path from $p$ to $q$, and a path from $q$ to $r$. Define $f$ so that it maps the path from $p$ to $q$ to the path from $q$ to $r$. We are then forced to define $f$ on the path from $q$ to $r$ to follow a certain path from $r$ to $p$, in order for $f(f(x))+f(x)+x=0$ to hold. Then we can define $f$ to take the path from $r$ to $p$ to the path from $p$ to $q$.

In this way (if we chose our paths well so that everything turns out to be well-defined), we can define $f$ on a loop that goes from $p$ to $q$ to $r$ and then back to $p$ such that $f(f(x))+f(x)+x=0$ for every point on the loop. If we have chosen our paths well, this loop will go around the origin once with monotonically increasing angle, so that for any nonzero $x\in\mathbb{R}^2$ there is a unique (continuously varying) $t>0$ such that $tx$ is a point of our loop. We can now extend $f$ to all of $\mathbb{R}^2$ by defining $f(x)=t^{-1}f(tx)$ for this value of $t$ (and $f(0)=0$). That is, we partition $\mathbb{R}^2$ into scaled copies of our loop, and define $f$ in the same way on each one. The equation $f(f(x))+f(x)+x=0$ will then hold on all of $\mathbb{R}^2$, since it holds on each scaled loop.


OK, now here's an explicit example if you really want one. Let us take $p=(0,1)$, $q=(1,0)$, and $r=(-1,-1)$. Our first path will be the line segment from $p$ to $q$, parametrized linearly. Our second path will be the line segment from $q$ to $r$, but parametrized nonlinearly (this will make our $f$ nonlinear). Explicitly, for $0\leq t\leq 1$, we define the following values of $f$: $$f((1-t)p+tq)=(1-t^2)q+t^2r$$ $$f((1-t^2)q+t^2r)=-(1-t)p-tq-(1-t^2)q-t^2r$$ $$f(-(1-t)p-tq-(1-t^2)q-t^2r)= (1-t^2)q+t^2r$$

The first equation says that $f$ maps the line segment from $p$ to $q$ to the line segment from $q$ to $r$, but with a nonlinear parametrization of the latter line segment (using $t^2$ instead of $t$). The second equation then defines $f$ on the line segment from $q$ to $r$ in the only possible way such that $f(f(x))+f(x)+x=0$ for any $x$ on the segment from $p$ to $q$. This means that $f$ maps the line segment from $q$ to $r$ to some quadratic path from $r$ to $p$. Finally, the third equation defines $f$ at points on the quadratic path so that $f(f(f(x)))=x$ for all the points where we have defined $f$ so far. This will make it so $f(f(x))+f(x)+x=0$ is also true on the line segment from $q$ to $r$ and on the quadratic path from $r$ to $p$.

We want to know that this loop we have defined by these three paths goes around $0$ once with strictly increasing angle (in particular, this means our quadratic path is injective and doesn't cross the line segments, so our definitions of $f$ cannot conflict with each other). It is clear that this is true for the two line segments; checking that it is also true for the quadratic path is just a messy computation. Plugging in the definitions of $p$, $q$, and $r$, we see that the quadratic path is parametrized by $$t\mapsto (t^2+t-1,2t^2-t-1).$$ To see the angle is increasing, we can define $$g(t)=\arctan\left(\frac{2t^2-t-1}{t^2+t-1}\right)$$ (for an appropriate branch of $\arctan$) and show $g'(t)>0$ for $0\leq t\leq 1$. This is just a computation: $g'(t)$ is a rational function in $t$, and with some work you can show its numerator and denominator are always positive for $0\leq t\leq 1$ (see http://www.wolframalpha.com/input/?i=derivative+of+arctan((2t%5E2-t-1)%2F(t%5E2%2Bt-1)), for instance). We can also check that the quadratic path does not loop all the way around the circle (for instance, since its second coordinate is always negative).

So this loop passes through every angle with respect to the origin exactly once. This means that for any nonzero $x\in\mathbb{R}^2$ there is a unique $t>0$ such that $tx$ is on the loop, and we can define $f(x)=t^{-1}f(tx)$ (and $f(0)=0$) as described above. Finally, this $f$ is nonlinear: for instance, $f(p)=q$ and $f(q)=r$ but $$f\left(\frac{p+q}{2}\right)=\frac{3q+r}{4}\neq \frac{q+r}{2}=\frac{f(p)+f(q)}{2}.$$


The computational part of Eric Wofsey's answer can be simplified a little bit ; in particular it can be done entirely by hand as I show below. Following this answer, let us start with the three points $p=(0,1),q=(1,0),r=(-1,-1)$. Let $b$ be a $C^1$ bijection $[0,1]\to [0,1]$, to be defined later, and let

$$ \begin{array}{lclcl} A(t) &=& (1-t)p+tq &=& (t,1-t) \\ B(t) &=& (1-b(t))q+b(t)r &=& (1-2b(t),-b(t)) \\ C(t) &=& -(A(t)+B(t)) &=& (2b(t)-(t+1),t+b(t)-1). \\ \end{array} $$ Then there is a unique bijection $f$, defined on the triangle $pqr$, such that $f$ cyclically permutes every triple $(A(t),B(t),C(t))$ for $t\in [0,1]$.

As shown in Eric's answer, the only problem is then to find a $b$ such that f is non-linear and such that the angle $(Or,OC(t))$ varies monotonically as $t$ goes from $0$ to $1$.

It is easy to see that $f$ is linear iff $b$ is the identity (indeed, $f$ cyclically permutes $p,q$ and $r$, and this suffices to uniquely define $f$ is $f$ is linear).

A little computation shows that the straight lines $(qr)$ and $(OC(t))$ intersect at $D(t)=(1-d(t))r+d(t)p$ where

$$ d(t)=\frac{2t-b(t)}{3t-3b(t)+1} $$

So it suffices to find a $b$ such that the derivative of $d$ is never zero on $(0,1)$. Another little computation shows that the numerator of $d'$ is (here $u$ and $v$ denote the numerator and denominator of $d$ respectively)

$$ N(t)=u'(t)v(t)-u(t)v'(t)=(3b(t)-1)b'(t)+2-3b(t) $$

If we try $b(t)=t^2$, we are lucky : $N(t)=3t^2-2t+2=3(t-\frac{1}{3})^2+\frac{5}{3}$ is never zero.