Are the rationals a closed or open set in $\mathbb{R}$?

In the usual topology of $\mathbb{R}$, $\mathbb{Q}$ is neither open nor closed.

The interior of $\mathbb{Q}$ is empty (any nonempty interval contains irrationals, so no nonempty open set can be contained in $\mathbb{Q}$). Since $\mathbb{Q}$ does not equal its interior, $\mathbb{Q}$ is not open.

The closure of $\mathbb{Q}$ is all of $\mathbb{R}$: every real number is the limit of a sequence of rationals, so every real number lies in the closure of $\mathbb{Q}$. Since $\mathbb{Q}$ does not equal its closure, it is not closed.

Naturally, since $\mathbb{Q}$ is not open, its complement is not closed; since $\mathbb{Q}$ is not closed, its complement is not open.

But this is in the usual topology. $\mathbb{R}$ can be endowed with lots of topologies, and it is certainly possible for $\mathbb{Q}$ to be open (or closed) in some of them. For example, in the discrete topology, where every subset of $\mathbb{R}$ is both open and closed, $\mathbb{Q}$ is both open and closed.

If the rationals were an open set, then each rational would be in some open interval containing only rationals. Therefore $\mathbb{Q}$ is not open.

If $\mathbb{Q}$ were closed, then its complement would be open. Then each irrational number would be in some interval containing only irrational numbers. That doesn't happen either. This one is harder to prove: showing that every interval on the line contains some rationals depends on the fact that the ordered field $\mathbb{R}$ is Archimedean.

Neither. Their interior is empty and their closure is the entire line.