Mean Value Theorem: Continuous or Defined?

The function $f:[a,b]\to R$ is continuous at $x=a$ if

$$\lim_{x\to a^+} f(x)=f(a)$$

Similarly $f:[a,b]\to R$ is continuous at $x=b$ if

$$\lim_{x\to b^-} f(x)=f(b)$$

The continuity at the interior points is two sided while the continuity at endpoints is one sided.

The other answers do a good job addressing the question from an analytic perspective. Topologically, there is an analogous explanation (which exists because metric spaces like $\mathbb{R}$ are also topological spaces, a more general class of objects):

Definition: A function $f:X \to Y$ between topological spaces is continuous if, for every open set $U \subset Y$, its preimage $f^{-1}(U)$ is open in $X$.

Note: When $X$ and $Y$ are metric spaces, this definition of continuity agrees with the epsilon-delta definition from analysis. The open sets in a metric space are unions of open "balls"; see here.

So user532874's concern can be expressed as follows:

Consider a purportedly continuous function $f:[a,b] \to \mathbb{R}$ and the open set $\Big(f(a) \! - \! \varepsilon, \ f(a) \! + \! \varepsilon\Big)$ for a small $\varepsilon >0$. If $f$ is continuous, then the preimage of this set should be an open set as well. But the preimage instead includes a half-open interval $[a,c)$ or even, if $f$ is constant, the entire closed interval $[a,b]$. Ostensibly, this preimage is not open: in particular, there doesn't seem to be any "wiggle room" around $x=a$. What gives?

And the answer is that, if $U$ is a subset of a topological space $X$ and we have a continuous function $g:U \to Y$, then the preimage of open sets in $Y$ will be open not with respect to $X$, but to $U$ which has been endowed with the subspace topology. Under this topology, the open sets of $U$ are $\{V \cap U \ | \ \text{V is an open set w.r.t. X}\}$.

In our scenario, we have $X = \mathbb{R}$ and $U = [a,b]$. Given the subspace topology, note that a half-open interval $[a,c)$ actually is "fully" open in $[a,b]$ because $[a,c) = [a,b] \cap (a \! - \delta, \ c)$ for any $\delta > 0$, and this interval $(a \! - \delta, \ c)$ is open in $\mathbb{R}$. Likewise, $[a,b]$ is open with respect to itself since $[a,b] = [a,b] \cap (a \! - \! \delta, \ b \! + \! \delta)$.

Using the definition of limit,"approaching" $a$ means that for any real number $\delta >0$ (no matter how "tiny"), we consider the elements $x\in [a,b]$ such that $|a-x|\le\delta$, and the similarly for $b$. This implies that this elements as close to $a$ (or $b$) as they might be, must be in the domain of $f$ in the first place, and as a result the limit exists if it exists from the side we can approach $a$ (or $b$). (Note that $f$ is indeed defined at $a$ and $b$ because they are in the domain of $f$ by hypothesis)

Hope this helps!