How can I prove it to be a well-defined binary operation?

There is no issue of "well-definedness": the elements of $G$ are real numbers, and for every real number $r$, $r-\lfloor r\rfloor$ is a real number that lies in $[0,1)$; thus, $\star$ is a function from $\mathbb{R}\times\mathbb{R}$ to $[0,1)$, and hence by restriction $\star$ is a function from $G\times G$ to $[0,1)\subseteq G$.

(Note: The next paragraph was written when the definition of $G$ was that $G$ included $1$; this has since been changed.)

However, as given, $G$ is not a group under the operation: note that if $(G,\cdot)$ is a group, then in particular for every $g\in G$ there exist $x,y\in G$ with $x\cdot y = g$. However, there are no elements of your $G$ which satisfy $x\star y = 1$, even though $1\in G$.

If, however, you change $G$ to be $G=\{x\in\mathbb{R}\mid 0\leq x\lt 1\}$, then the set is indeed a group (It is, in fact, isomorphic to $\mathbb{R}/\mathbb{Z}$). Note that $0$ is a two-sided identity, since if $x\in G$ then $\lfloor x\rfloor = 0$; and that if $x\neq 0$ is in $G$, then $1-x\in G$ and $x\star(1-x) = 0$. And trivially, since addition of reals is commutative, $x\star y = y\star x$.

So the only thing that needs to be proven is associativity.

The key is to note that for all real numbers $r$ and all integers $n$, $$\lfloor r-n\rfloor = \lfloor r\rfloor - n.$$

So, if $x,y,z\in [0,1)$, then: $$\begin{align*} (x\star y)\star z &= \Bigl( x+y - \lfloor x+y\rfloor\Bigr)\star z\\ &= \Bigl( x+y-\lfloor x+y\rfloor + z\Bigr) - \lfloor x+y - \lfloor x+y\rfloor + z\rfloor\\ &= x+y+z - \lfloor x+y\rfloor -\Bigl( \lfloor x+y+z\rfloor - \lfloor x+y\rfloor\Bigr)\\ &= x+y+z - \lfloor x+y+z\rfloor;\\ x\star(y\star z) &= x\star\Bigl( y+z - \lfloor y+z\rfloor\Bigr)\\ &= x + y + z - \lfloor y+z\rfloor - \lfloor x+y+z-\lfloor y+z\rfloor\rfloor\\ &= x+y+z - \lfloor y+z\rfloor - \bigl( \lfloor x+y+z\rfloor - \lfloor y+z\rfloor\bigr)\\ &= x+y+z - \lfloor x+y+z\rfloor\\ &= (x\star y)\star z. \end{align*}$$

You can also go the longer route and consider the possibilities of $x+y\lt 1$, $y+z\lt 1$, $x+y+z\lt 1$; or $x+y\lt 1$, $1\leq y+z\lt 2$ and $1\leq x+y+z\lt 2$; $x+y\geq 1$, $y+z\geq 1$, and $1\leq x+y+z\lt 2$; and $x+y\geq 1$, $y+z\geq 1$, and $2\leq x+y+z\lt 3$. But the observation above is much easier.

HINT $\ $ The only difficult part is proving associativity. Denote the fractional part of a real by $\rm\: f(r) = r - \lfloor r\rfloor = r\ mod\ 1\:.\:$ Note that $\rm\:x\star y = f(x+y)\:.\:$ We CLAIM $\rm\ f(x+f(y)) = f(x+y)\:$
Proof: let $\rm\:f(y) = r,\ y = r + n,\ n\in \mathbb Z\:.\:$ Then the claim is equivalent to $\rm\:f(x+r) = f(x+r+n)\:$
which is true: adding an integer $\rm\:n\:$ doesn't alter the fractional part. Hence applying claim twice

$$\rm\ x\star(y\star z)\ =\ f(x+f(y+z))\ =\ f(x+y+z)\ =\ f(f(x+y)+z)\ =\ (x\star y)\star z $$

NOTE $\ $ The same proof yields associativity of integer addition $\rm\:mod\ m\:,\:$ with $\rm\:f(k) = k\ mod\ m\:.\:$ The essence of the matter will be much clearer after you learn about quotient groups. Then you will see that this group is simply the real "circle group" $\rm\:\mathbb R\ mod\ \mathbb Z\:$ of reals modulo $1\:.\:$ That the group laws are preserved in every modular image becomes a triviality from this general standpoint.

I'm expanding Bill Dubuque's note:

Forget about fractional parts for the moment. Call two real numbers $x$, $y$ equivalent if $y-x\in{\mathbb Z}$. Denote the equivalence class of $x\in{\mathbb R}$ by $[x]$ and the set of all equivalence classes by $F$. The definition $$[x]+[y]\ :=\ [x+y]$$ defines addition uniquely on $F$ (check this!), $[0]$ is the neutral element, and the addition inherits from ${\mathbb R}$ the familiar properties: commutativity, associativity and existence of (additive) inverses. Therefore $F$ is an abelian group.

Now each equivalence class $[x]\in F$ contains exactly one element of your set $G:=[0,1[\ \subset{\mathbb R}$, namely the (well defined!) number $x-\lfloor x\rfloor$. It then follows that $G$ is a complete set of representatives for $F$. So instead of dealing with classes you can just add their representatives, and that's what has been set up in your problem.