What is the mathematical distinction between closed and open sets?

Let's talk about real numbers here, rather than general metric or topological spaces. This way we don't need notions of Cauchy sequences or open balls, and can talk in more familiar terms.

We define that a set $X \subset \mathbb{R}$ is open if for every $x \in X$ there exists some interval $(x-\epsilon,x+\epsilon)$ with $\epsilon > 0$ such that this interval is also fully contained in $X$.

An example is the inverval $(0,1) =\{x \in \mathbb{R} : 0 < x < 1\}$. Note that this is an infinite set, because there are infinitely many points in it. If you choose a number $a \in (0,1)$ and let $\epsilon = \min\{a-0, 1-a\}$ then we can guarantee that $(a-\epsilon,a+\epsilon) \subset X$. The set $X$ is open.

A set $X$ is defined to be closed if and only if its complement $\mathbb{R}- X$ is open. For example, $[0,1]$ is closed because $\mathbb{R}-[0,1]= (-\infty,0)\cup(1,\infty)$ is open.

It gets interesting when you realise that sets can be both open and closed, or neither. This is a case where strict adherence to the definition is important. The empty set $\emptyset$ is both open and closed and so is $\mathbb{R}$. Why? The set $[1,2)$ is neither open nor closed. Why?

The question you asked has answers at various levels of sophistication. You really want to be thinking about intervals rather than finite sets of points, and think also about regions in the plane. Here is one way of looking at things.

A closed set is one which contains all its limit points - so when you are working with closed sets you can be confident abut taking limits because you know that the limit points exist.

An open set can be thought of as one in which every point is an interior point, at least that is a useful guide when you are thinking about sets on the line or in the plane. So the idea is that if you pick a point in an open set, you have enough points close to it in the set to work with, and that there are no points close to it which are outside the set (every point in an open set has a neighbourhood of points wholly within the open set). This cashes out particularly when thinking about continuous functions - the classic epsilon-delta definition is essentially talking about the relationships between nearby points.

These ideas can be considerably generalised and made precise as part of the machinery of topology. Note it is possible to have a set which is both open and closed -- the whole of the real line for example -- or to have a set that is neither open nor closed, such as the set of all rational numbers.

The basic properties of closed and open sets are not the only useful things about them. For example a closed and bounded subset of the real line has a useful property called "compactness", which enables us to reduce some infinite problems to finite ones, and hence get better results.

A real-valued continuous function on a closed bounded subset of $\mathbf R^n$ always has a maximum and a minimum. This is not true for open subsets.

Example: the function $\dfrac1{x(1-x)}$ has no maximum on $(0,1)$.