Various examples of non-representable functors
First a comment about $\alpha$. $\newcommand\Hom{\operatorname{Hom}}\newcommand\Set{\mathbf{Set}}\newcommand\CRing{\mathbf{CRing}}\newcommand\Ring{\mathbf{Ring}}\newcommand\Top{\mathbf{Top}}\newcommand\Vect{\mathbf{Vect}}$The "natural" direction for a natural transformation is $\alpha : \Hom(A,-)\to F$, because by the Yoneda lemma, such natural transformations are equivalent to the elements of $FA$. Thus when we say an object $A$ represents a functor $F$, we usually give a pair $(A,r)$ with $r\in FA$, such that the natural transformation induced by $r$, $f\in \Hom(A,B) \mapsto F(f)r \in F(B)$ is a natural isomorphism.
Note that when we present representing objects this way, we won't run into case (3) in your question. I'm not aware of any time that this ever happens, and I'm also failing to produce an artificial example.
Let's start with some classic examples of representable functors on the category of commutative unital rings. As mentioned in an answer that was just written, adjoint pairs give a large range of representable functors, and some of these examples will fall under that category, but I think we can gain intuition from looking at some in more detail.
Let $\CRing$ be the category of unital, commutative rings.
Ex 1. In $\CRing$, the polynomial rings $\Bbb{Z}[x_1,\ldots,x_n]$ represent the functor that sends a ring $S$ to the $n$th Cartesian power of the underlying set, $S^n$ (the set of $n$-tuples of elements of $S$). The representing $n$-tuple is $(x_1,\ldots,x_n)\in (\Bbb{Z}[x_1,\ldots,x_n])^n$. This is because a map $\phi:\Bbb{Z}[x_1,\ldots,x_n]\to S$ is determined precisely by the $n$-tuple $(\phi(x_1),\ldots,\phi(x_n))$. (This comes from the free-forgetful adjunction)
The basic observation in (1) allows us to build a wide range of examples, and is one of the most common ways of constructing representing objects for functors on rings. The next few examples should give the idea here. I won't fully explain them for the sake of brevity though.
Ex 2. Let $F(S) =S^\times$ be the functor that sends a ring to its set of units. This is represented by $(\Bbb{Z}[x,y]/(xy-1),x)$. This ring is often written as $\Bbb{Z}[x,x^{-1}]$
Ex 3. Let $n$ be a fixed natural number, and let $F(S)=\{x\in S : x^n = 0\}$. Then $F$ is represented by $(\Bbb{Z}[x]/(x^n),x)$.
Ex 4. Generalizing the previous examples, if $F$ can be described as the set of solutions of some polynomial equations in the ring $S$, then it is representable. I.e., if there are polynomials in $\Bbb{Z}[x_1,\ldots,x_n]$, $f_1,\ldots, f_m$ such that $$F(S) = \{(s_1,\ldots,s_n)\in S^n : f_i(s_1,\ldots,s_n) = 0 \text{ for all $i$}\},$$ then $F$ is represented by $(\Bbb{Z}[x_1,\ldots,x_n]/(f_1,\ldots,f_m),(x_1,\ldots,x_n))$.
This is usually how we construct representing objects for a functor from $\CRing$ to $\Set$, we look to see if we can put $F(S)$ into bijection with the set of solutions of some polynomial equations. A classic example of this is the construction of the Lazard ring.
Counterexample 1. This brings me to my first counterexample, still focusing on the category of commutative rings. Let $F(S)=\{ s\in S : s^n = 0\text{ for some $n\in\Bbb{N}$}\}$ be the functor that sends a ring to the set of all of its nilpotent elements. $F$ is not representable. Why?
Well, suppose we had a representing object, $(R,x)$ with $x\in F(R)$ such that ring maps $\phi : R\to S$ are in bijective correspondence (via $\phi \mapsto \phi(x)$) with the set of nilpotent elements in $S$. Well, since $x\in F(R)$, it is nilpotent. This means that there is some natural number $n$ such that $x^n=0$ in $R$. However, then if we consider $S=\Bbb{Z}[y]/y^{n+1}$, $y$ is nilpotent, but we cannot define a ring map $\phi : R\to S$ such that $\phi(x)=y$, since if we could, then we would have $0=\phi(x^n) = \phi(x)^n =y^n\ne 0$.
Another perspective is to think about what the representing object would have to look like. If we let $\newcommand\Nil{\operatorname{Nil}}\Nil_n(S)=\{s\in S: s^n=0\}$, we have $F(S) = \bigcup_n \Nil_n(S)$. We know from (Ex 3) that $\Nil_n$ is representable, represented by $\newcommand\Z{\mathbb{Z}}(\Z[x]/(x^n),x)$. We can write this as $$ F(S) = \operatorname{colim}_n \Nil_n(S) = \newcommand\colim{\operatorname{colim}} \colim_n \CRing(\Z[x]/(x^n),S) \overset{?}{=} \newcommand\of[1]{\left({#1}\right)} \CRing\of{ \lim_n \Z[x]/(x^n), S }, $$ where the limit of the rings is taken over the quotient maps $\Z[x]/(x^{n+1})\to \Z[x]/(x^n)$. The question mark over the equality is because, while we're taking directed limits/colimits here, which have better properties than generic limits/colimits, this is still generally false, and it turns out to be false in our case. The limit of these rings does exist. It's the power series ring, $\Z[[x]]$. However $x$ is no longer nilpotent in $\Z[[x]]$, so it cannot be our representing object.
However, if we enlarge our category to commutative, unital topological rings, and give ordinary rings the discrete topology and $\Z[[x]]$ the $(x)$-adic topology, then $x$ is topologically nilpotent ($\lim_{n\to\infty} x^n = 0$), and $\Top\CRing(\Z[[x]],S)$ where $S$ is an ordinary commutative unital ring with the discrete topology is actually in bijection with $F(S)$.
As to how this all relates to your cases, we basically have that for certain candidate representing objects, e.g., $\Z[x]/(x^n)$, we have an isomorphism between $\CRing(\Z[x]/(x^n),S)$ and $F(S)$ exactly when $S$ has no elements which have higher nilpotency. I.e., if $F(S)=\Nil_n(S)$. This is case (2) in your question.
Case 1
Also I should note that case (1) in your question is weird, because in most natural categories/functors, you can construct something that works for at least one object.
However, here's a construction which will produce an example of case (1). Let $P$ be a poset, regarded as a category. Note then that any representable functor must have the set $F(x)$ be either empty or a singleton, since the hom sets in $P$ are $$ P(x,y) = \begin{cases} \{*\} & x\le y \\ \varnothing & \text{otherwise.} \end{cases} $$
Then we define a functor $F(x) = \{ z \in P : z \le x\}$, where $F(x\le y)$ is the inclusion map $F(x)\hookrightarrow F(y)$, since if $z\le x$, then $z\le y$. Now if for example we take $P=\Z$, then $F(x)$ is infinite for all $x$, so $F$ cannot be representable, and moreover we find ourselves in case (1), where no component of any natural map from a representable functor to $F$ can be an isomorphism.
Let me give one or two possible observations that I happen to use quite often when dealing with representability to answer the following question of yours:
"When you find a functor in the wild, what about it leads you to think "hmmm this is probably going to be representable because ..."?"
1) Some functors you are interested in might arise as a (post-)composition with the (here covariant) $\text{Hom}$-functor. These are for example representable. These include for example making basechanges (in some suitable category where this makes sense) before applying the $\text{Hom}$-functor.
2) A big class of representable functors are right adjoints. Right adjoints are representable and often one has an idea in mind when defining a functor that it is part of an adjunction. That is at least to hope for since that is a very useful property. This makes for example forgetfull functors representable if you have a notion of a corresponding free functor, i.e. free objects in your category.