What does the "closed over"/"closed under" terminology mean exactly and where did it come from?

In an ordinary mathematical context I would never say that an operation $\mathcal O$ is closed over a set $A$; I consider that an error for ‘$A$ is closed under $\mathcal O$’. The latter terminology can be properly applied very generally. For starters, if $A \subseteq S$, $n \in \omega$, and $f:S^n \to S$, ‘$A$ is closed under $f$’ means precisely that for every $\langle a_1,\dots,a_n \rangle \in A^n$, $f(a_1,\dots,a_n) \in A$.

This is the most common algebraic usage, I think, but it’s just the beginning. For instance, the non-negative integers $n$ can be replaced by any ordinal $\alpha$, and the operation need not be defined on all members of $S^\alpha$. Suppose that $X$ is a topological space and $A \subseteq X$. ‘$A$ is closed under (the operation of taking) limits of convergent sequences’ then means that if $\langle a_n:n \in \omega \rangle$ is a sequence of points in $A$ that converges as a sequence in the space $X$, its limit point is actually in $A$. Here $\alpha = \omega$, and the operation is defined only for those elements of $X^\omega$ that converge in $X$. This example at least starts to show the relationship between the general notion and that of topological closedness.

Moreover, the terminology is still used when the input to the operation isn’t ordered: it’s perfectly correct to say that a family $\mathcal A$ of sets is closed under (taking) finite intersections, for instance, meaning that if $\mathcal F$ is any finite subfamily of $\mathcal A$, $\bigcap \mathcal F \in \mathcal A$. If $S$ is the underlying set, the operation is $\mathcal O:[(\mathcal P(S)]^{< \omega} \to \mathcal P(S):\mathcal F \mapsto \bigcap \mathcal F$, and $\mathcal A \subseteq \mathcal P(S)$ is closed under it because $\mathcal O$ maps $[\mathcal A]^{< \omega}$ into $\mathcal A$. (Here $[X]^{< \omega}$ denotes the set of finite subsets of $X$.)

It would be a bit difficult to formulate an exhaustive formal definition of the usage, and I’m not at all sure that it would be particularly helpful; it seems more useful to present a variety of examples showing the flexibility of the usage.