What does Hartshorne mean by an (open) affine subset?

The explanation for this is hidden in the statement of exercise I.3.5, which begins with the following sentence:

"By an abuse of language, we say that a variety 'is affine' if it is isomorphic to an affine variety."

The upshot is that the property of "being affine" is an intrinsic property independent of the embedding, which is a very nice thing to have.

In my edition of Hartshorne this question is answered near the beginning of 1.4:

Before giving this result, we need a couple of lemmas which say that on any variety, the open affine subsets form a base of the topology. We say loosely that a variety is affine if it is isomorphic to an affine variety.

So "open affine subset" means "open subset which is isomorphic, as a variety, to an affine variety." Somewhat confusingly, it should be read as "affine (open subset)"; that is, "affine" modifies "open subset." "Affine subset" is meaningless because Hartshorne has not defined what a morphism from an arbitrary subset of a variety to a variety is, which means he hasn't defined what it means for an arbitrary subset of a variety to be affine. (It's possible to give such a definition using the language of scheme theory.)

You ask in the comments:

But this seems silly -- what does it mean for a set (that is not already a variety) to be isomorphic to a variety?

This is a subtle point which is often not properly addressed. The answer is that Hartshorne defines a variety to be any of an affine, quasi-affine, projective, or quasi-projective variety. An open subset of an affine variety is a quasi-affine variety, and in 1.3 Hartshorne defines morphisms between varieties, which lets you write down a definition of a morphism from a quasi-affine variety to an affine variety, and hence lets you define what it means for two such things to be isomorphic.

The simplest example to keep in mind is the punctured affine line $\mathbb{A}^1 \setminus \{ 0 \}$, which is not an algebraic subset of $\mathbb{A}^1$. But it is open and so it's a quasi-affine variety, and as a quasi-affine variety it's isomorphic to an affine variety, namely the hyperbola $\{ xy = 1 \}$ in the affine plane $\mathbb{A}^2$. So it "is" affine. This is all cleared up by learning more about localization.

(On the other hand, the punctured affine plane $\mathbb{A}^2 \setminus \{ 0 \}$ is not affine in the sense that, as a quasi-affine variety, it is not isomorphic to an affine variety.)