How to define limit operations in general topological spaces? Are nets able to do this?

The notion of limit is well-defined for any topological space, even non-metric ones.

Here is the correct definition: let $f : X \rightarrow Y$ be a function between two topological spaces. We say that the limit of $f$ at a point $x \in X$ is the point $y \in Y$ if for all neighborhoods $N$ of $y$ in $Y$, there exists a neighborhood $M$ of $x$ in $X$ such that $f(M) \subset N$.

But note that the sequential characterization of limit ($f$ tends to $y$ in $x$ iff for every sequence $(x_n) \rightarrow x$, one has $f(x_n) \rightarrow y$) is not true in a general topological space. It is true if $X$ is metrizable.

A net is a function from an directed set $(I, \le)$ (say) to a space $X$.

$f: I \to X$ converges to $x$ iff for every open set $O$ that contains $x$ there exists some $i_0 \in I$ (depending on $O$, in general) such that for all $i \in I, i \ge i_0$ we know that $f(i) \in O$.

The point $f(i)$ is often denoted by subscript: $x_i$. This subscript an be any member of the directed set $I$. A sequence is the special case where $I=\mathbb{N}, \le)$ (in its standard order). The definition is non-metric in that we use open sets containing $x$ (not open balls) but for metric spaces it suffices to check this for open balls $B(x,\varepsilon), \varepsilon>0$ as these form a local base at $x$.

I think that $\lim_{x \to a} f(x)$ can be defined by considering all nets $n$ on $X\setminus \{a\}$ that converge to $a$, and if all those nets have the property that $f \circ n$ is a net in $Y$ converging to the same $b \in Y$, then this $b$ is called the limit of $f$ as $x$ tends to $a$.

If $X$ has a topology induced from a metric, e.g. then we can restrict ourselves to sequences instead of general nets in the above characterisation.

The general definition of continuity is as follows: let $X$ and $Y$ be topological spaces and $f:X \to Y$ be a map. $F$ is continuous if the inverse image of any open set is open. $f$ is continuous at a point $x$ iff for every open set $V$ containing $f(x)$ there exists an open set $U$ containing $x$ such that $f(U) \subset V$.

Continuity of $f$ is equivalent to the following: whenever a net $(x_i)_{i \in I}$ converges to some point $x$ we have $x$ we have $(f(x_i))_{i \in I}$ converges to $f(x)$. Similarly, $f$ is continuous at a point $x$ iff whenever a net $(x_i)_{i \in I}$ converges to $x$ we have $x$ we have $(f(x_i))_{i \in I}$ converges to $f(x)$.