functor $\texttt{Nil}: Ring \longrightarrow Set$ is not representable

I assume that by "ring" you mean "commutative ring." Suppose $\text{Nil}$ is represented by some commutative ring $N$. Then $\text{id}_N \in \text{Hom}(N, N) \cong \text{Nil}(N)$ must be the "universal nilpotent" $n \in N$: that is, it is a nilpotent with the property that it maps to every other nilpotent in every other commutative ring $R$ under a (unique) homomorphism $N \to R$. (Every representable functor works this way: see universal element. This is $a$ in your work.)

But there can't be a universal nilpotent, because any nilpotent element must be nilpotent of some particular degree $k$. And if $n^k = 0$ then $n$ can't map to nilpotents of degree larger than $k$. For a bunch of variations on this argument see this blog post. In your work you attempt to map the universal nilpotent to something which is not a nilpotent at all.

---

Alternatively although similarly, you can argue that $\text{Nil}$ doesn't preserve infinite limits. (A representable covariant functor preserves all limits.) In fact it already fails to preserve infinite products, as follows: consider the product

$$R = \prod_{k \in \mathbb{N}} \mathbb{Z}[x]/x^k.$$

Then each $x \in \mathbb{Z}[x]/x^k$ is nilpotent but the product element $\prod x$ is not.

Qiaochu Yuan has given good advice in general, but I'd already started writing this answer, and it's more a review of what you've written.

You've almost got it. It's all correct up to your last paragraph. The problem is that you chose $B=\Bbb{Z}[x]$ and $b=x$, but the universal property of $a$ only guarantees that there exists a map $g:A\to B$ with $g(a)=b$ when $x$ is nilpotent. However in this case it is not, since $\Bbb{Z}[x]$ is a domain.

Instead, observe that if $a^n=0$ for some $n$, which must exist since $a$ is nilpotent, then consider $B=\Bbb{Z}[x]/(x^{n+1})$. Then there must exist $g:A\to \Bbb{Z}[x]/(x^{n+1})$ with $g(a)=x$, but then $0=g(a^n)=g(a)^n=x^n\ne 0$. Contradiction.