Can endpoints be local minimum?
Actually, the question is settled by reading the definition you provided carefully:
A function $f$ has local maximum value at point $c$ within its domain $D$ if $f(x)\leq f(c)$ for all $x$ in its domain lying in some open interval containing $c$.
I.e., the points $x$ for which the condition must hold are required to both be in the open interval and in $D$.
To see that $(1,1)$ is a local maximum, consider the open interval $(0, 2)$. If $x \in (0, 2)$ and $x$ is also in the domain $[1,\infty)$, then $1 \le x < 2$. Now $g(x) = x^2-4x + 4 = (2 - x)^2$. So $g(1) = (2 - 1)^2 = 1^2 = 1$, but if $x > 1$, then $0 < 2 - x < 1$, so $0 < (2-x)^2 = g(x) < 1$. So for $x$ in the open interval $(0,2)$ and also in the domain $[1,\infty)$, we have that $g(x) \le g(1)$.
I think fundamentally the comments are right, and you should speak with your teacher to confirm definitions and expectations. But there's also a point to make about topology here, which could justify the book's definition and answer as consistent.
The definition of local maximum you gave is:
A function $f$ has a local maximum at point $c$ within its domain $D$ if $f(x) \leq f(c)$ for all $x$ in its domain lying in some ** open ** interval containing $c$.
If you interpret this as saying that the interval can come from $\mathbb{R}$, and is not restricted to $D$, then you have no problem, as others have pointed out. But like you I am thinking about being restricted to $D$ and my instinct is to think only about intervals in $D$. This can still be ok, if we just alter our interpretation of "open" a little bit (in a natural way)...
Now, whenever we say "open" we're really saying "open with respect to ** insert topology here ** ." A lot of the time it's obvious from context or the textbook has established a practice of contextual implication, but in this case (without knowing your book) I'd argue there are two reasonable interpretations:
- We might be talking open intervals with respect to the standard topology on $\mathbb{R}$ (which is what you've probably been using in your class), but
- since we're restricting our attention to a domain $D \subset \mathbb{R}$, it's also pretty normal to talk about a different topology, called the subset topology on $D$ (induced by the standard topology on $R$).
In the subset topology on $D \subset \mathbb{R}$ (induced by the standard topology), a set $S$ is open if and only if $S$ is the intersection $D \cap X $, with $X$ open in $\mathbb{R}$ with respect to the standard topology on $\mathbb{R}$.
We're often more interested in the subset topology than the usual topology on the whole space just because of situations like the one you're in, in which a definition doesn't work quite like you expect when $D \not= \mathbb{R}$.
So let's work with a slightly different definition of local maximum:
A function $f$ has a local maximum at point $c$ within its domain $D$ if $f(x) \leq f(c)$ for all $x$ in its domain lying in some interval $I$ containing $c$ such that $I$ is open with respect to the subset topology on $D$.
Now back to your case. Let $D = [1, \infty)$. For any $a > 1$, we have that $$[1,a) = D \cap (-a,a)$$ Since $(-a,a)$ is open in $\mathbb{R}$ with respect to the standard topology, $[1,a)$ is open in $D$ with respect to the subset topology on $D$. This intuitively makes sense, because if you were an ant walking on $f(D)$, when you came to $f(1)$ you'd have nowhere to go but down.