Why do "Author's Rights" in "Consent to Publish" agreements allow pre-print publications?

Short answer: Pre-prints are not a big threat to the business models of journals and there are scientific norms that mean that journals are expected to permit pre-prints.

Longer Answer:

Why do universities and others pay for journals when some of the articles in these journal subscriptions are available via pre-print servers and other means?

• Readers want access to the copy of record supplied by the journal.
• Readers want access to every single article, not just the subset that are available as pre-prints.
• Readers want efficient and consistent access.
• Readers want an attractively formatted PDF or HTML copy.
• Although not a pre-print server, it's also worth noting that universities respect copyright and therefore do not see services like scihub that provide more comprehensive free access as legitimate.

Why do many journals allow authors to share pre-prints?

• They don't see it as a big impost on their business model given the points above.
• Many journals have an embargo period on post-prints (i.e., pre-prints that have been updated based on the feedback provided by journal peer-review). These are presumably designed to encourage purchase of journal subscriptions.
• Some journals may appreciate the importance of academic dissemination even where the potential readers do not have a subscription.
• In some fields, pre-prints are very entrenched to the point that if a journal did not allow pre-prints, they might be seen poorly by that field and receive fewer high quality submissions.
• More generally, there is competition amongst journals, and allowing pre-prints is one small way that journals can compete for submissions.
• There has also been an ongoing debate about the business models of journals. Journals are receiving pressure from various parties to permit these forms of distribution.
• Also, pre-print servers are generally non-commercial. I have seen some publishers ban positing on ResearchGate because they are for-profit and serve Ads alongside pre-prints.

What am I missing?

That we are the geese who are laying golden eggs for the publishers. Their business model is to do little and to get paid huge sums of money for it. Scientists do essentially all the work: the writing, peer review, and the editorial work. All that publishers really do is copyediting, and quite frankly they often do a mediocre job of that.

If publishers forbade researchers from posting preprints, then researchers would post preprints anyway. If publishers tried to enforce such a restriction, then researchers would stop submitting there.

The academic publishing industry is an example of economic rent seeking, generating profits without creating wealth. The balance is precarious, and publishers understand that it is in their economic interests not to disrupt it.

That's an interesting example of two different Nash equilibria. If all top journals in a field allow to publish preprints, then one journal/publisher disallowing it would simply lead to no-one publishing there. If the norm is to forbid it, then one journal allowing it will probably just decrease its revenue (it would attract more good papers, but that's not directly monetizable).

Same on the scientists' side: if everyone else publish on Arxiv, then publishing my papers in a journal that disallows it would simply diminish the visibility of my work. If the norm is to embargo/paywall publications, then by not following it I just cut myself off most good journals.

Historically, different fields ended up in different equilibria; for that reason, some use arXiv, some don't. This is also en example of a situation where a regulation, e. .g, mandating "green" open access by some big funding agencies, could move the situation out of the "bad" equilibrium. Whether it's what happening is another question.