Is $\mathbb{N}$ impossible to pin down?

If you are a Platonist, or in any other sense believe that there is a "true" set of natural numbers, then you have pinned them down. If you believe that there is one concrete universe of sets, and suppose it even satisfies the axioms of $\sf ZF$, then that's universe $\omega$ can be thought of as the one true set of natural numbers.

But if you are not a Platonist. Whether you are a formalist, or supporting a multiverse approach, or maybe you just don't care enough to believe that one thing is true or another, then there is indeed a slight problem because we can switch between models of $\sf PA$ and models of $\sf ZF$, and thus get different "true" natural numbers.

But the point is this, in my opinion, that when we are set to work and we take $\sf ZF$ to be our foundational theory, then we fix one universe of set theory that we work in. And being foundational this universe cannot disagree with the notion of natural numbers coming from the logic outside of it (so in particular it is going to have other models of $\sf ZF$ within it). Then the natural numbers are the $\omega$ of that universe.

When you are done working with this universe you throw it in the bin, and get another when you need to. Or you can save that universe in a scrapbook if you like.

The reason we can do this, or better yet, the reason that people don't care about foundational problems in many places throughout mathematics, is that we use properties which are internal to that set, and therefore are true in any set with those properties, regardless it being the "true" or "false" set of natural numbers. If you write the sentence $\lnot\rm Con(\sf PA)$, and shown it to be independent of $\sf PA$ then you have shown that first-order logic is insufficient to determine all the true properties of the natural numbers. But those that it does determine are enough for a lot of things, and most people don't mind switching to second-order in some of the cases. In which case the above argument fails to hold.

There are three closely related concepts:

• The collection of natural numbers

• The collection of (correct) formal proofs

• The collection of finite sequences from a fixed finite alphabet (for example, the collection of finite sequences of 0s and 1s).

Forgetting about formal systems for a moment, if we can pin down just one of these three concepts, the other two are also going to be pinned down - but it is very difficult to define any of these three concepts without referring to the others. That difficulty shows up in particular when we try to define the natural numbers in formal systems, or try to formalize proofs within a formal system such as ZFC.

There is no reason that we must interpret this as saying that the categoricity of $\mathbb{N}$ is "illusory" - we could also view it as saying that effective formal systems are simply never strong enough to prove all number-theoretic truths. This deficiency of formal systems only affects $\mathbb{N}$ in settings where $\mathbb{N}$ is defined using a formal system. Many mathematicians feel they already understand what $\mathbb{N}$ is before they learn anything about formal systems, however.

Yes, you are correct, at least in some sense.

But rather than thinking of "the natural numbers", I prefer to think more along the lines of, for each (adequate) set-theoretic universe, there is a particular model $\mathbb{N}$ having a specific, useful relationship to the universe containing it.