Can $\mathcal O_X$ be recognized abstract-nonsensically?

I think you are a little confused about what your characterization of $R$ does, and this causes problems as you generalize to sheaves.

There is no characterization of $R$ as an element of the category of $R$-modules via abstract nonsense - this is because tensoring by any rank $1$ locally free module is an autoequivalence of this category. Instead, what you have is a construction of the element from the category. But there is no way of telling which element of the category actually corresponds to this ring you have constructed.

If you want to characterize $\mathcal O_X$, rather than construct it, you face the same problem - there are autoequivalences which cause trouble, so you need extra structure.

If you want to construct $\mathcal O_X$, you face a different problem - $\mathcal O_X$ is a much more complicated object than just a ring. It might be hard to construct all of $\mathcal O_X$ without already knowing something about how the open sets of $X$ relate to the category. For instance one could know, for each open set, the subcategory of sheaves that are pushforwards from that open set. Then I believe one can recover $\mathcal O_X(U)$ as the endomorphisms of the identity functor on the subcategory of sheaves that are pushforwards from $U$.

One certainly can't recover it just from the category and the topology on $X$, because there can be automorphisms of the topological space $X$ that do not fix $\mathcal O_X$.

So the only hope to construct $\mathcal O_X$ from just the category of $\mathcal O_X$-modules is to first construct the topological space $X$ from the category and then construct the sections on it. One would then not be able to construct the isomorphism of this topological space with a concrete $X$ unless given other data on $X$.

An idea for how to do this is that, for each closed set $Z$, the category of sheaves supported on $Z$ is a Serre subcategory, and the endomorphisms of the identity functor on the Serre quotient by that category should be the sections of $\mathcal O_X$ on the complement of $Z$. So it would be sufficient to characterize these subcategories among all Serre subcategories.

As Will points out, the best you can hope for is to recognize being a line bundle. Also, I find you're a bit vague about what category of sheaves you want work in; I'm thinking about coherent sheaves, which is an appropriate analogue for finite dimensional $R$-modules. As we'll see below, I'll also want to assume my scheme is locally of finite type (over a field, or other Jacobson ring, such as $\mathbb{Z}$) EDIT: nfdc23 also points out I should assume reduced.

I have an answer for this is which is probably ultimately equivalent to Heinrich's but which I like better (perhaps the difference is he's thinking of quasi-coherent sheaves?). The only simple objects in the category of coherent sheaves are the skyscraper sheaves at the different closed points. Thus, we can actually reconstruct the closed points of the scheme. You can also define the support of a sheaf by which skyscrapers it has maps to, and reconstruct the Zariski topology by defining a closed set to one that appears as the support of a coherent sheaf. A coherent sheaf is a line bundle if and only if the Hom space to every skyscraper sheaf is 1-dimensional.

EDIT: Why is this so? A coherent sheaf is a line bundle if and only every stalk is free of rank 1 over the local ring at each point. A module over a local ring is free of rank 1 if and only if the rank of its residue at the unique closed point and the generic point is 1. So, it's enough to check that you have residue of rank 1 at every point (including non-closed ones).

So, if you have rank 1 at all closed points, by semi-continuity, you have rank 0 or 1 at every point, and the set where you have rank 1 is closed. Thus, you must have rank 1 everywhere if the closed points are dense (if not, the structure sheaf on their closure will be a counterexample). This occurs whenever you are locally of finite type (as discussed here: https://math.stackexchange.com/questions/615709/is-the-set-of-closed-points-of-a-k-scheme-of-finite-type-dense). Presumably outside finite type, one can modify the construction a bit to include the non-closed points.