Why don't we take clopen maps as morphisms of the category of topological spaces?

Well, the obvious reason is that such a definition would lack the properties of continuous maps that we'd expect. For example, for the Sierpiński space consisting of $\{\bot,\top\}$ with open sets $\{\{\},\{\top\},\{\bot,\top\}\}$ the only "clopen map" is the identity function. If we defined continuous map as "clopen map" then, in this case, not even constant functions would be continuous. Still, this tells you that this is a bad definition for "continuous", but not how you might be led to the right one, particularly without a moment of insight.

We can be more systematic though. For an algebraic structure, like a group, the structure consists of a collection of operations operating on a set. For example, inversion is a unary group operation $G \to G$. Given an operation $\mathtt{i}_G:G\to G$ and a function $\varphi:G\to H$ which, let's say, is a bijection, then a very natural notion of "preserving" $\mathtt{i}$ is that $\mathtt{i}_G(g) = \varphi^{-1}(\mathtt{i}_H(\varphi(g)))$ which clearly leads to $\varphi(\mathtt{i}_G(g)) = \mathtt{i}_H(\varphi(g))$ which doesn't require $\varphi$ to be a bijection.

The structure for a topological space $X$ is topology $\mathcal{T}_X$ which is a subset of $\mathcal{P}(X)$, or, equivalently, a subset of $X\to\mathbf{2}$. We could take the view that each open subset $U_X\in\mathcal{T}_X$ is an "operation", $X \to \mathbf{2}$. The natural notion is, given a function $f : X \to Y$ is that an operation is preserved via $U_X(x) = U_Y(f(x))$ for a suitable $U_Y\in\mathcal{T}_Y$. In particular, this states that given an open subset of $Y$, $U_Y$, we get an open subset of $X$ via $U_Y \circ f$. Viewing $U_X$ and $U_Y$ as subsets, the preservation condition is exactly that $f^{-1}(U_Y) = U_X$. Of course, this is different from the algebraic case as we don't have a fixed set of operations that are rigidly mapped one to the other. At any rate, the "backwardness" isn't so surprising when you consider the contravariance of $\mathcal{P}(X)$ (i.e. $X \to \mathbf{2}$) as a functor in $X$.

Of course, this doesn't explain what operations topologies should be closed under.

The description of continuity in terms of preimages of open sets/closed sets is primarily of computational convenience. A more intuitive definition is that a function $X\xrightarrow{f}Y$ is continuous if it maintains closeness of points to sets in the sense that $f(\overline U)\subseteq\overline{f(U)}$ for any $U\subseteq X\xrightarrow{f}Y$. Here $\overline U$ is the closure of a subset $U\subseteq X$ and consists of all points of $X$ that are close to $U$, i.e. a topology on a set $X$ can be interpreted as the specification of a (well-behaved) closeness relation between points of $X$ and subsets of $X$, and continuity as the preservation of this closeness relation.

With a little bit of thought, it should be easy to see that the notions of closed and open maps, while preserving closed or open subsets, do not necessarily preserve the closedness relation; conversely, maintaining the closedness relation does not mean that closed sets or open sets are preserved. So to answer your question, we don't use clopen maps because they preserve a different structure from the one we are trying to capture. (In fact, I think in this context the notions of open and closed are somewhat artificial, perhaps merely artifacts of the attempt to describe the closedness relation).

To address the comparison to groups, not all structures are given by operations, structures can also be given by relations.

Because then $f : \mathbb{R} \to \mathbb{R}$, $x \mapsto e^x$ would not be a morphism (its image is not closed). Recall that almost every notion in mathematics is motivated by examples, and this includes the definition of a category and explicit examples of categories. You don't want to just play around with the axioms, you want to study the examples you are actually interested in. And a definition of a morphism of spaces which does not include the exponential function is certainly not natural.

Another approach to topological spaces is provided by Kuratowski spaces. Here, one has a relation $x \prec A$ between points $x$ and subsets $A$ of the underlying set, which is supposed to mean that $x$ lies in the closure of $A$. This relation $\prec$ has to satisfy some axioms. Then, a map $f$ is continuous if and only if it preserves this relation $\prec$: $$x \prec A \Rightarrow f(x) \prec f(A)$$

You might also want to have a look at frames; here the morphisms are really algebraic homomorphisms as you suggest. Frames resp. locales are a generalization of topological spaces. This is also called "pointless topology".