When the definition of a set starts to matter in category theory
To consolidate a clarification from the comments:
In mathematics as normally practiced, there is no such thing as a "definition" of the notion of set. There are various axiomatic theories one can use to reason about objects which we call "sets" (or think of as "set-like"), such as ZFC, ETCS, type theory, etc. But in these theories, "set" is typically a primitive, undefined notion. This is much as in classical Euclidean geometry, where the terms "point", "line", etc. are undefined, but we have well-defined rules for reasoning about them. The only place you would "define what a set is" is if you were constructing a model of one of these theories -- then you have to define what the model is going to say a "set" is, relative to whatever metatheory you're using.
But I think the larger question is -- "when does one have to worry about set-theoretic details when doing category theory?" The question is a bit ill-defined, but maybe there are a few things to say anyway:
- The most prominent "set-theoretic consideration" in category theory is that of size. When doing category theory, it's important to develop a sense for which "set-like things" one considers are actually sets, and which ones are "too big to be sets" -- i.e. are actually proper classes [1]. Two theorems of basic category theory come immediately to mind -- both due to Freyd:
Any small category with all small products is a preorder. This theorem tells us that if we want to work with completeness conditions in category theory, then we really have to live with the fact that there's a difference between large and small.
The adjoint functor theorem crucially requires that certain collections appearing in the hypotheses be small (i.e. that they be sets rather than proper classes). The standard counterexample is: for each cardinality $\kappa$, let $G_\kappa$ be a simple group of cardinality $\kappa$, and define $F: Grp \to Set$ to be the functor $G \mapsto \prod_\kappa Hom(G_\kappa,G)$. This functor preserves limits, but has no left adjoint.
- There are also "set-theoretic considerations" which are really just an annoyance. For instance, for most purposes it doesn't matter whether, say, the category $Ab$ of abelian groups is skeletal or not (because $Ab$ is equivalent to any of its skeleta, which are all isomorphic -- there is nothing special about $Ab$ here). But sometimes you might want to do something like take the product of all countable abelian groups. When you do this, it technically matters -- if $Ab$ is skeletal, this is a perfectly well-defined small abelian group, whereas if $Ab$ is defined as usual in ZFC say, there is a proper class of countable abelian groups (though just a small number of isomorphism classes). So to perform this construction in usual set theory, you should probably say something like "the product of one copy from each isomorphism class of countable abelian groups" and then proceed as you would have otherwise.
[1] In this post, I will use the language of a set / class distinction to talk about size. There are also other approaches, most notably the approach of using one or more Grothendieck universes. In the Grothendieck universe approach, there are no proper classes -- everything is a set. It's just that some sets are "small" while others are "large".
I didn't have time to write up a proper answer on initially seeing your question and much of what I have to say has been said in the comments and Tim's answer, but I'll still offer some specifics on things mentioned in the comments.
The definition of a set is, as Tim Campion pointed out, the axioms of whatever set theory you're working in. They do not so much define a set as describe a primitive notion that behaves like what we think sets should behave like, with different axiomatizations giving rise to differing notions of set.
These differing axiomatizations do have an impact on category theory because they change the behavior of ${\bf Set}$, and consequently the behavior of presheaf categories, and they also impact our ability to manipulate large categories. As mentioned by Ingo Blechschmidt in the comments, Mike Shulman has an excellent paper surveying some of these consequences. I will summarize some of them here, but I highly recommend you check out his paper.
A striking result referenced in the Shulman paper is due to by Colin McLarty, establishing that the NF axiomatization of what a set is yields a ${\bf Set}$ that isn't Cartesian closed.
In ZFC we really only run into issues if we want to manipulate large categories as a whole, for example ${\bf Set}$ or ${\bf Group}$, which are not actually objects in ZFC since they're proper classes. We can get around this with shenanigans about formulas the metalanguage, but anyone looking for an integrated and 'natural' treatment of large categories on level footing with small ones will be disappointed in this setting.
NBG is a conservative extension of ZFC (meaning it doesn't prove anything about sets that ZFC can't) which does allow proper classes to be real objects in the theory, but we still run into some discomfort when dealing with large categories. NBG manages to be conservative over ZFC by restricting it's comprehension axiom to only apply to sets, not proper classes -- in practical terms, as Mike points out in his paper linked above, this means (for example) that we can't prove by induction that a large category $\mathcal{C}$ has an $n$-fold Cartesian product $\mathcal{C}^n$. We can get around this by constructing it directly as the category of functions from $n$ into $\mathcal{C}$, but the unavailability of canonical proof methods like induction is troubling.
MK is a non-conservative extension of ZFC, essentially NBG but with full class comprehension allowed so we have access to all of the standard proof tools for large categories. This new theory can prove things ZFC can't, like the consistency of ZFC, and is thusly strictly stronger in a meaningful sense. MK also has its own serious issues when working with large categories -- we can't define the category of functors between two large categories, and this applies to NBG as well.
Using full MK further suggests that we try to look at the category of classes, since they're really the category of collections we want to work with right? And bam, once again we're back to a situation where we have to play games in the metalanguage, or conservatively extend/step up the consistency strength of our theory. This leads mathematicians to situations like Grothendieck universes, where it's always possible to step up to the next universe if we need to talk about 'all the somethings' in the current universe. This is equivalent to working in ZFC plus an axiom asserting the existence of an inaccessible cardinal.
All the extra baggage of universes or inaccessibles is still somewhat of a sledgehammer for the problem at hand, though; all we want is for large categories to 'be like small categories' in enough ways that we can carry out all the constructions we care about with large categories, but inaccessibles or Grothendieck universes also have a plethora of other consequences (like the need to juggle universes*). A solution to these problems comes in the form of reflection principles, which are essentially axioms asserting that proper classes look enough like sets that we don't have to soil ourselves when they appear, but don't endow them with enough independence to give rise to a whole hierarchy of universes we need to ask questions about. All of this is discussed at length in the Shulman paper referenced above, with additional references therein.
First paper: Shulman, Mike. Set theory for category theory. arXiv:0810.1279v2 [math.CT]
Second paper: McLarty, Colin. Failure of Cartesian Closedness in NF. J. Symbolic Logic 57 (1992), no. 2, 555--556. https://projecteuclid.org/euclid.jsl/1183743976
*As Tim points out in the comments, how many universes we have to juggle when taking this route is up to us. Skilled jugglers may use an infinite number, while those new to the approach may use only two.