Is anybody researching "ternary" groups?

One problem with the idea is that the most obvious generalization to ternary operations really adds nothing new:

Proposition: Let $f:G\times G\times G:\to G$, and for brevity write $[abc]$ for $f(a,b,c)$. Suppose that there is an identity element $1_G\in G$ such that $[a1_G1_G]=[1_Ga1_G]=[1_G1_Ga]$ for all $a\in G$. Suppose further that the operation is associative in the following sense: $$\big[[abc]de\big]=\big[a[bcd]e\big]=\big[ab[cde]\big]$$ for all $a,b,c,d,e\in G$. Then there is an associative binary operation $\otimes$ on $G$ such that $[abc]=a\otimes b\otimes c$ for all $a,b,c\in G$, and $1_G$ is the $\otimes$-identity.

The proof is easy. Define $\otimes:G\times G\to G$ by $a\otimes b=[ab1_G]$. Then

$$\begin{align*} (a\otimes b)\otimes c&=[(a\otimes b)c1_G]=\big[[ab1_G]c1_G\big]=\big[ab[1_Gc1_G]\big]=[abc]\\ &=\big[[abc]1_G1_G\big]=\big[a[bc1_G]1_G\big]=[a(b\otimes c)1_G]=a\otimes(b\otimes c)\;, \end{align*}$$

and $a\otimes 1_G=[a1_G1_G]=a=[1_Ga1_G]=1_G\otimes a$ for all $a,b,c\in G$. Note that this does not require any kind of generalized commutativity for the ternary operation.

I remember noticing this as an undergraduate in the late 60s. My roommate was looking at a less obvious generalization of the associative law. The generalization, as I recall, was $$\big[[abc]de\big]=\big[a[bde][cde]\big]\;.\tag{1}$$ The idea is that if $\otimes$ is a binary operation on a set $G$, one can think of each element $a\in G$ as defining a function $f_a:G\to G:x\mapsto a\otimes x$, and associativity of $\otimes$ is then the statement that $$\operatorname{Comp}(f_a;f_b)=f_{f_a(b)}\;,\tag{2}$$ where $\operatorname{Comp}$ is the composition operator. In the ternary setting think of $a\in G$ as defining a function $$f_a:G\times G\to G:\langle b,c\rangle\mapsto [abc]\;;$$ then $(1)$ becomes

$$\operatorname{Comp}(f_a;f_b,f_c)=f_{f_a(b,c)}\;,$$

generalizing $(2)$.

If I remember correctly, this approach produced more interesting structures, but I no longer remember the details.

As Brian pointed out, the idea of ternary groups isn't that interesting, since ternary operations with identity can be shown to reduce to successively applied binary operations.

But there are some ternary group-like structures you might find interesting, and that might satisfy some of the curiosity that led you to ask this question in the first place. Someone named Dave Barber has been studying ternary quasigroups and has generated Cayley tables for a lot of them, along with names to describe their unique properties.

In constructing a ternary quasigroup, starting from a set of just 4 elements yields 55,296 possible ternary operations to choose from. Starting with 5 elements gives you 2,781,803,520 possible ternary quasigroups, and 6 elements gives 994,393,803,303,936,000 possible ternary quasigroups. Of course some of these can be can be expressed in terms of boring old binary operations, but many of them cannot! That's pretty exciting to me.