Understanding the support of a function

Will add to this later, but just to give a start:

Your $f$ is supported everywhere. To see that it's supported away from the origin, for example, note that if we throw out the origin then what we get is isomorphic to $\mathbb{A}^1 - \{(x)\}$. Now, $f$ isn't zero in the corresponding ring, which is a domain, so it has non-zero germ at each point.

There are two concepts at work here and it's natural to be confused. We think about functions as taking values and vanishing in the residue fields and this is what defines our topology. But the support is where the germ is non-zero. Maybe it's good to think about how, in plain old analysis, the $C^\infty$ functions $0, x, x^2$ have value zero at the origin but only $0$ has a trivial germ there and indeed $x, x^2$ are supported, in the old sense, on the whole line!

I'll try to think about whether there is some statement about vanishing [in the sense of values]. The trouble is that $y$ takes on the value $0$ at every point but it contains more information than the zero section — it ought to live somewhere. Probably you can say something in be reduced case.

We approach the problem by investigating the stalks of this scheme and deducing which global functions may be supported at each point.

First observe that the prime ideals of $A$ are $(Y)$ and $\mathfrak{m}_a=(Y,X-a)$ for each $a \in k$, and that $A_{\mathfrak{p}} \cong k[X,Y]_{\mathfrak{p}}/(Y^2,XY)_{\mathfrak{p}}$ for each prime $\mathfrak{p}$. Thus if we think of an element of $A$ as being given by a polynomial $f(X,Y)$, then $f$ is supported at every point if it is not divisible by $Y$*. But any element of $A$ divisible by $Y$ may be represented by $f(X,Y) = cY$ for some $c \in k$. But then this is supported exactly at $\mathfrak{m}_0$, unless $c = 0$ in which case it is supported nowhere.

Putting this together, we have three possibilities for $\operatorname{Supp}(f)$:

$i)$ $\phi$ if $f = 0$

$ii)$ $\{\mathfrak{m}_0\}$ if $f$ is a nonzero multiple of $Y$

$iii)$ $\operatorname{Spec(A)}$ if $f$ is not divisible by $Y$

We can in fact interpret this in terms of reducedness. We can think of reducedness as "remembering extra information", in this case the reducedness of the scheme at the origin is the scheme "remembering that it is part of the plane" at the origin (Vakil says something similar when he introduces this example). This is rather vague, but making this notion precise is part of the point of section $5.5$. One way to think about it is like this: consider the function $Y$ on two domains, the plane and the $X$-axis. This function on the plane vanishes precisely on the $X$-axis and nowhere else, so the germ of $Y$ as a function on the plane vanishes nowhere (since a germ vanishing at a point means that the function vanishes on some open set about this point). However, as a function simply on the $X$-axis, $Y$ is identically zero, so the germ of $Y$ vanishes everywhere. We think of $\operatorname{Spec}(A)$ as being the $X$-axis that "remembers that it lives in the plane at the origin", the germ of $Y$ unexpectedly fails to vanish at the origin, because there the scheme remembers that $Y$ doesn't vanish identically on any open set about the origin in the plane, even though these open sets aren't contained in our scheme at all.

*This may not be immediately clear, but we have that $(Y^2,XY)_{\mathfrak{p}} \subset (Y)_{\mathfrak{p}}$, and so any $f$ whose germ vanishes at $\mathfrak{p}$ must be divisible by $Y$. In fact, the containment above is actually an equality unless $\mathfrak{p} = \mathfrak{m}_0$, since if this doesn't happen, $X$ is a unit in the localised ring.

Adding to TTS answer I think I can now solve Vakil's exercise entirely.

We want to find all points $\mathfrak p$ such that there exist one $f \in A$ such that the support of $f$ is $\mathfrak p$. This amounts to the following two requirements:

(1) f is nonzero in $A_\mathfrak p$

(2) f is zero in $A_\mathfrak q$ for all $\mathfrak q ≠ \mathfrak p$

but (2) is equivalent to the following

(2') there is an element $a \in \mathfrak p$ s.t. f is zero in $A_a$.

Now our $A$ was by definition $k[x,y]/(y^2,xy)$, so we see that all $\mathfrak p \in A$ contains $(y)$. We have the general point $\mathfrak p_0$, which is the support of 1, and a prime ideal for each point on the line $\mathbb A$, i.e. a prime ideal of the form $(x-c, y)$. We will now show that the only $c$ for which the corresponding prime is a support of some $f$ is $c = 0$.

Given an $f$ the only way that in some local ring it will be zero is if in that local ring an element $g$ is inverted s.t. $fg = 0$. In particular, $g$ is a zero-divisor. Since $y$ is in all prime ideals of $Spec (A)$, we only have $x$ left. So we have to require that our $\mathfrak p$ is such that $x$ is inverted in all other $A_\mathfrak q$. It follows that $c$ must be $0$.