When is every direct product of a ring also a free module?
By the results in the paper of O'Neill, any $F$-ring is semiprimary, so the Jacobson radical $J(R)$ is nilpotent (thus $J(R)=\operatorname{nil}(R)$ if $R$ is commutative) and $R/J(R)$ is semisimple. Commutative semisimple rings are finite products of fields, so in particular they are zero-dimensional. Since quotienting out the nilradical doesn't change the dimension, any commutative semiprimary ring is zero-dimensional and thus any commutative semiprimary domain is a field.
To give an example of a commutative ring that is an $F$-ring, but not a domain, note that by example 3.7 and theorem 4.2 in O'Neill's paper, if $R/J(R)$ is simple Artinian (so a field in the commutative case), then $R$ is an $F$-ring iff $R$ is right Artinian, so we can take for example $k[x]/(x^2)$ for a field $k$.
Let me supplement Mathein Boulomenos's answer with some concrete elementary arguments for some simple examples. First, note that if $R=S\times T$ is a product of two rings, then every $R$-module $M$ is the direct sum of the submodules $(1,0)M$ and $(0,1)M$, which can be thought of as just an $S$-module and a $T$-module. In particular, then, a product $R^X$ is the direct sum $S^X\oplus T^X$. If $S$ and $T$ are $F$-rings, then $S^X$ and $T^X$ are free over $S$ and $T$, respectively. As long as $S^X$ and $T^X$ have the same rank, we can then pair up the copies of $S$ and $T$ in $S^X$ and $T^X$ to write $R^X$ as a direct sum of copies of $R$, and so $R^X$ is free over $R$. In particular, if $S$ and $T$ are countable, then $S^X$ and $T^X$ will always have the same rank for cardinality reasons.
Thus a product of two countable $F$-rings is an $F$-ring. In particular, any finite product of countable fields is an $F$-ring. (Note that the cardinality assumption here actually matters, and so describing which products of fields are $F$-rings involves some nontrivial set theory. For instance, if $S$ is a countable field and $T$ is a field whose cardinality satisfies $|T|^{\aleph_0}>2^{\aleph_0}$, then $T^\mathbb{N}$ has larger rank over $T$ than $S^\mathbb{N}$ has over $S$ and so $R^\mathbb{N}$ is not free and $R$ is not an $F$-ring!)
Now let us consider the example $R=k[x]/(x^2)$ where $k$ is a field. Let $M=R^X$, and choose a basis $B$ for $M/xM=k^X$ over $k$. I claim that when we lift $B$ to a subset $C$ of $M$, we get a basis for $M$ over $R$. First, multiplication by $x$ gives an isomorphism from $M/xM$ to $xM$, so the submodule generated by $C$ contains $xM$ and thus is all of $M$ since $B$ generates $M/xM$. Second, the linear independence independence of $B$ means that any linear relation between elements of $C$ must have all coefficients divisible by $x$. Dividing all the coefficients by $x$ and using the isomorphism $M/xM\cong xM$, we would then get a linear relation between elements of $B$, so the coefficients must actually be $0$. Thus $C$ is linearly independent.
This line of argument can be generalized to show that any artinian local ring is an $F$-ring, by induction on length. Let $R$ be an artinian local ring with maximal ideal $m$ and let $x\in R$ generate a minimal nonzero ideal. Let $M=R^X$, choose a basis for $M/mM$ and lift that basis to $M$. By induction on length, the lifted basis is a basis for $(R/(x))^X$ over $R/(x)$. Then you can argue as above using the fact that multiplication by $x$ gives an isomorphism $M/mM\to xM$ that the lifted basis is in fact a basis for $M$ over $R$.