Is it true that the order of the group is a power of $2$ if every element has order $2$?

For finite groups it's definitely true. Suppose $|G|$ was divisible by an odd prime $p$. By Cauchy's Theorem, $G$ contains an element of order $p$

I should add, clearly this does not hold when $G$ is an infinite group, since every non-identity element of $\mathbb{Z}_2^{\infty}$ has order $2$.

No group consists only of elements of order $2$, because the identity always has order $1$ :-)

Leaving this nitpick aside, it's well-known that a group of exponent $2$ is abelian because $$ a(ba)b = (ab)^2 = e = a^2 b^2 = a(ab)b $$ (followed by canceling $a$ from the left and $b$ from the right), so if the group is finite it's isomorphic with $({\bf Z/2\bf Z})^m$ for some $m$.

JasonM's argument is still preferable because it generalizes to arbitrary primes: a finite group of exponent $p$ has order $p^m$ for some $m$, else some other prime $q$ divides the group order, and then Cauchy's theorem produces an element of order $q$.

I would like to propose a more direct answer without using Cauchy's theorem. First note that any group $G$ with only order 2 elements apart form the identity is abelian. Consider the minimum number of generators required to generate the group, say there are $n$ of these. Then every element of the group must be a product of these $n$ generators. Since the group is abelian, the order of the generators within a product is inconsequential and there are at most $\sum_{i=1}^n \binom{n}{i}$ of such products. Adding the identity element itself, this makes at most $2^n$ elements, so $|G|\leq 2^n$. But in fact all of the products must be distinct (and different from the identity) since otherwise we can always rearrange (multiplying by inverse generators and so forth) such that one supposedly independent generator is in fact a product of other generators. Thus $|G|=2^n$, where $n$ is the minimum number of generators of $G$.