Why isn't finitism nonsense?
It's not known that modern set theory is consistent; in fact, by the Incompleteness Theorem, we can't ever have a system of axioms that we can prove is consistent. Which means that the only condition we can rely on for determining whether a set of axioms is "right" is whether or not it produces absurd results.
Under $ZFC$, we have different sizes of infinity - there are sets which are larger than the set of natural numbers in a precise sense. We also have a lot of weirdness involving the Axiom of Choice - for example, with the Axiom of Choice, a theorem of Banach and Tarski states that a hollow sphere can be disassembled into five pieces and then reassembled (without stretching, tearing, or otherwise deforming the pieces) into two spheres that are both identical to the first one in both size and shape. But the Axiom of Choice simply states that given a set of sets, we can "choose" one element from each set - which seems intuitively true.
A finitist's perspective on $ZFC$ is often that results like the hierarchy of infinite cardinals and the Banach-Tarski paradox are absurd - that they should count as contradictions, because they patently disagree with the intuitive picture of mathematics. The sensible conclusion is that one of the axioms of $ZFC$ is wrong. Most of them are intuitively obvious, because we can demonstrate them with finite sets - the only one we can't is Infinity, which states that there exists an infinite set. So a finitist's conclusion is to reject the Axiom of Infinity. Without that axiom, $ZFC$ becomes purely finitistic.
Now, many finitists are happy to stop here. But some are bothered by the fact that we still have an infinite collection of natural numbers; the infinite still "exists", in a sense, and gives the opportunity for the above weirdnesses to arise in the same way. So some people (including some mathematicians) subscribe to ultrafinitism and insist that there are only finitely many numbers at all. One ultrafinitist mathematician I know defines the largest integer to be the largest integer that will ever be referenced by humans.
Among mathematicians, ultrafinitists are much rarer than simple finitists. Finitists generally agree with you that "unboundedness" is a natural idea - it's essential, for example, in the definition of a limit. But they would go on to insist that this is just a formalism - that a limit, for example, is just a statement of eventual behavior, involving only finite numbers. So $\infty$ isn't an object, it's just a shorthand. This is (at least to my mind) more mathematically defensible than ultrafinitism.
EDIT: Since a lot of people seem to be having a hard time with my first sentence, I thought I'd clarify. The Incompleteness Theorem states that we cannot have a set of axioms powerful enough to express arithmetic and still be able to prove its consistency within the system. The reason I didn't include this phrase above is because it unnecessarily weakens the point. Any axiom system intended to codify all of mathematics defines the idea of "proof"; if, say, $T$ is intended to underlie all of mathematics, then by "proof" we must mean "proof in $T$". With such a $T$, we can say that $T$ cannot be proven consistent at all; because by Incompleteness, any proof of the consistency of $T$ would not be a proof from inside $T$, but $T$ is supposed to be powerful enough that all proofs are proofs from inside $T$. Thus: we can't ever have a system of axioms for mathematics that we can prove is consistent - full stop.
To add to Reese's excellent answer, I will say first that I don't consider myself a finitist, but I can understand why finitists postulate as they do—even the ultrafinitists.
First, to quote the Wikipedia article already cited in a comment (emphasis added):
Even though most mathematicians do not accept the constructivist's thesis, that only mathematics done based on constructive methods is sound, constructive methods are increasingly of interest on non-ideological grounds. For example, constructive proofs in analysis may ensure witness extraction, in such a way that working within the constraints of the constructive methods may make finding witnesses to theories easier than using classical methods. Applications for constructive mathematics have also been found in typed lambda calculi, topos theory and categorical logic, which are notable subjects in foundational mathematics and computer science.
Now my own explanation of ultrafinitism has less to do with belief than practicality. But first, a discussion of ideas themselves.
Numbers, ultimately, are ideas, as are every other element in mathematics.
You say (emphasis added):
Are there some (maybe arguably) good mathematical reason to deny the existence of ∞ or is it just a philosophical attitude? The concept of unboundedness seems pretty natural to me, so what could be a reason to avoid it?
This is inherently a subjective position, and a perfectly valid one: "I can think of this idea, so how can someone say this idea doesn't exist?"
Of course it would be ludicrous to claim that the idea of infinity doesn't exist. You just thought of it (thought about it), didn't you?
I'll be an ultrafinitist for a moment and explain the position.
It's not that infinity doesn't exist as an idea, it's that you cannot point to an infinity anywhere in the real universe. Anything you point to is necessarily finite, or you couldn't point it out or demonstrate it.
Mathematics is all (all) based on working with symbolization of real or abstract data. You're dealing with ideas, fundamentally, and ways of representing those ideas to resolve, communicate about, or pose problems—again, either real or abstract.
Please don't be so attached to a single system for ideas and their symbolization that you fail to recognize that other ideas may exist outside of that scope.
You criticize ultrafinitists for failing to include the concept of an infinity in their abstractions and symbolizations. Very well, why is it that your own mathematics fail to include the concept of "certainty"? Or "knowledge"? Or "co-existence" (the same number having two different values at the same time)? Or how about "time" itself, since that is not included in mathematics?
If you can work with your mathematics and get results that work, or even just that you find interesting, fine. If I can work with my mathematics and get different results than you, but they work for me (produce a desired result when applied to the real world), great.
But this is all more general, covering the broad sweep of differences of mathematical ideas.
To answer your precise question, and provide the "arguably good" mathematical reason to omit consideration (not "deny the existence") of the infinite in a mathematical framework, it is:
If you omit everything that cannot be directly observed, and abstract only that which can be observed, your results will apply uniformly to the observable universe.
This conclusion itself can only be demonstrated by observation of the observable universe—it cannot be theoretically evolved. It itself is separate from the approach of theoretically evolving a set of ideas, so it cannot be measured by the yardstick of theoretical postulation of ideas.
Chew on that for a bit. :)
Even if I'm speaking as an ultrafinitist, I would still say there is one factual infinity:
The possible different ideas that can be conceived of and posed by the human mind is infinite.
But that doesn't make the idea of an infinity inherently superior to the idea of no infinity. ;)
Leopold Kronecker was one of the leading mathematicians at the end of the 19th century. Kronecker disagreed sharply with contemporary trends toward abstraction in mathematics, as pursued by Cantor, Dedekind, Weierstrass, and others. Specifically, Kronecker rejected the notion of completed infinity. Many of today's mathematicians are so used to set theory being "the foundation" that they have difficulty relating to Kronecker's point to begin with. Kronecker's ideas were close to but not identical to the modern constructivist ideas; thus Bishop for one arguably did accept actual/completed infinity, since early on in his book he speaks matter-of-factly about functions $f$ from $\mathbb N$ to $\mathbb N$. The difficulty we have of relating to Kronecker's viewpoint has to do with our training. Recently Yvon Gauthier tried to explore Kronecker's position; see in particular his
Gauthier, Yvon. Towards an arithmetical logic. The arithmetical foundations of logic. Studies in Universal Logic. Birkhäuser/Springer, Cham, 2015
and
Gauthier, Yvon. Kronecker in contemporary mathematics, general arithmetic as a foundational programme. Rep. Math. Logic No. 48 (2013), 37–65.
Gauthier argues in particular that many applications Kronecker worked on can be handled without superfluous infinitary assumptions that merely clutter the picture.