How dangerous are set-size assumptions?
For what it's worth, the reason I made that comment was because when I gave talks expressing skepticism about the philosophical basis of ZFC, I would often get the reaction "but as long as it's consistent, what's the problem?" Some version of this attitude is also quite prevalent in the philosophical literature on the subject. I wanted to make the point that just being consistent isn't good enough: ZFC might well be consistent yet still prove, for example, that some Turing machine halts when in fact it does not. If you're a skeptic about sets but not about numbers, this could be a concern.
As for the "safety" of assuming ZFC is arithmetically sound, it is clear that as long as ZFC is consistent, no weaker system could prove that it is unsound. So for predicativists like me, who generally work in much weaker systems than ZFC, it seems unlikely that we could ever establish unsoundness. In that sense I'd say it's pretty "safe" to assume it is sound.
However ... I regard the Peano axioms as expressing intuitively evident truths about the natural numbers, so I have no problem affirming Con(PA). Whereas I regard ZFC as what you get when you realize that full comprehension leads to inconsistencies, so you replace it with a hodgepodge of instances of comprehension which appears to be weak enough to block any paradoxes. (Edit. In case my meaning isn't clear: the axiom of pairing is an instance of full comprehension, the axiom of unions is an instance of full comprehension, the axiom of power sets is an instance of full comprehension ... Basically what you're doing is throwing in a bunch of unrelated instances of "for any property $P$ there is a set of all $x$ such that $P(x)$.") It seems to me quite ad hoc and unmotivated, so although we might (or might not) be confident that ZFC is consistent, there is little reason to expect it to be (or not to be) arithmetically sound.
(I should add that I regard the "iterative conception" which is supposed to justify ZFC as utterly unconvincing. Sets are said to be built up in stages, and then it is parenthetically added that they are, of course, timeless abstract objects so they aren't really "built", nor do they really "appear" in "stages" --- it's all just a "metaphor". Thus "any conviction that the iterative conception may carry is made to depend on metaphorical details that are dismissed as inessential to it." (I. Jane))
This answer repeats some of the material in other answers but I think it is not entirely redundant.
As you seem to have realized, the assertion that a theory is consistent is a much weaker statement than the assertion that the theory does not prove false arithmetical statements (or is "arithmetically sound", to use the standard terminology). So the answer to your question about whether it could be the case that ZFC + universes is consistent but not arithmetically sound is yes, but for perhaps a trivial reason: ZFC + universes could be consistent and yet ZFC itself could fail to be arithmetically sound. Probably what you really wanted to ask was, if ZFC is arithmetically sound, and ZFC + universes is consistent, does it follow that ZFC + universes is arithmetically sound? The answer is no. For example, it is conceivable that under these hypotheses, ZFC + universes could prove $\neg$Con(ZFC). (EDIT: This was a typo. I meant that ZFC+universes could prove $\neg$Con(ZFC+universes). Thanks to Noah Schweber for catching this—see the comments.)
Any feeling that universes are a "safe" assumption must come from a direct assessment of the universes axiom itself. The reasons are similar to the reasons that we have for believing that ZFC is arithmetically sound.
The ability to remove Grothendieck universes from the proof of Fermat's Last Theorem cannot be deduced from some simple abstract fact like "Fermat's Last Theorem has low quantifier complexity." No meta-theorem of this sort is known. Eliminability must come from careful examination of the nitty-gritty details of the proof.
Your question about whether Con(PA) is philosophical or mathematical needs clarification. Offhand, it sounds like a false dichotomy to me, predicated on dubious assumptions about the meaning of the words "philosophical" and "mathematical." Can you define precisely what you mean by a "mathematical" assumption? Are you implicitly assuming that "mathematical" assumptions are rock-solid while "philosophical" ones are not? If so, that in itself is a dubious assumption IMO.
There's a lot going on in this question; let's break it down:
Is the assumption Con(PA) a philosophical one, and not a mathematical one?
I suppose it depends on what you mean by "the assumption Con(PA)":
If you're writing a mathematical proof, then either the foundations you're assuming prove Con(PA) or they don't, and it's a mathematical question which is the case. You'll be mathematically justified in assuming Con(PA) in the former case. In the latter case, there are contexts where "assuming Con(PA)" is mathematically justified -- for example, if you're trying to prove a statement of the form $Con(PA) \Rightarrow P$ for some $P$, then you're allowed to argue by assuming Con(PA) and proving $P$. But probably that's not what you mean.
If you're just asserting "PA is consistent" in a non-mathematical context, then your statement requires some philosophical unpacking to make precise, and moreover to determine to what extent it is justified. In this sense, it's a philosophical statement.
What principles lead us to believe that "Universes" is a safe assumption, whereas "¬Con(PA)" is not safe, regarding what we believe is "true" arithmetic?
There is an extensive philosophical literature discussing justifications for large cardinal axioms. You might start here. I'm not an expert, but I'm not aware of arguments which specifically argue that one should accept the arithmetic consequences of large cardinal axioms without arguing that one should accept the axioms outright. I would be interested to see such an argument. I once asked this question with related motivations.
I think the relevant thing to say is that I've never heard of similar sorts of justifications for $\neg Con(PA)$, and I don't expect I will.
(Next, repeat this question regarding the axiom of power set.)
Restrictions on powersets are studied in predicative mathematics. For some discussion of predicativism, one might start here or here. For some arguments for impredicativism, one might start here.
Is any theory that interprets PA "safe", as long as it is consistent with PA, and PA+Con(PA), and any such natural extension of these ideas?
By "safe", I take it that you continue to mean that the theory's arithmetic consequences are "true". There is a large body of literature discussing truth in mathematics. You seem to be particularly interested in the hierarchy obtained by passing from $T$ to $T + Con(T)$ iteratively. This hierarchy appears to be discussed from a philosophical perspective here. I think most logicians would probably agree that ascending this hierarchy adds relatively little consistency strength. And I'm not aware of serious attacks on the common-sense idea that if one believes $T$ is true, then one should also believe that $T + Con(T)$ is true.
Again, I'm not an expert, but it seems to me that most of the time when trying to justify stronger axioms from weaker assumptions, one appeals to some sort of reflection principle. This concept seems to be relevant to several of the questions you're asking.
You might also be interested in Reverse mathematics, where the "strength" of a mathematical theorem is calibrated by finding just how weak of a foundational theory can prove the theorem. One can interpret this as finding how "safe" a theorem is -- if you can prove it in a weaker theory, then it will continue to be true even if stronger theories are inconsistent, or simply "false".