Difference between initial and terminal objects in a category

that have an outgoing morphisms to every set (except the empty set)

That's the key part. The requirement is that there be outgoing morphisms to every set full stop. The existance of one set where this breaks down is enough to undermine it. That's why, indeed, the initial object in Set is already well-defined without even requiring uniqueness: the empty set is the only set that has an outgoing arrow to the empty set.

Meanwhile, every non-empty set has incoming arrows from truely every set including the empty one, but only for one-element sets is this arrow unique.

Here's the direct quote from the book: "The initial object is the object that has one and only one morphism going to any object in the category." Notice the only one part.

BTW, I was also careful not to say "any other object," because there also is a unique morphism from the initial object to itself: it's always the identity.