Can a UK University prevent you from graduating due to an outstanding library fine?

This is called an administrative encumbrance, it is indeed very standard.

The general notion is that if you owe the university money, then the administration will deny you things. These things may include: unit results, the ability to graduate, exam timetables, or the ability to enrol in new units. In what was certainly a bug, students at my wife's university were even denied access to the payment webpage while under-encumbrance -- which did not help them get paid.

There are often many ways to get encumbered, depending on the university and country. As well as library fines, there are often things like amenities fees, student union fees, parking fines, tuition costs.

Is it legal? Probably. You almost certainly agree to something along the lines of "I will suffer the effects of being encumbered if I fail to pay any fees or fines, and may not be allowed to graduate", as part of your enrolment or financing paperwork/online form tickbox. Is it a legally binding contract, though? Who knows, we will probably never find out as the cost of a lawyer to take it to court and actually prove it through contest is far more than the cost of just paying it. I would bet it is legal, as universities tend to have teams of lawyers on retainer.

Now the real question is will a <£10 fine actually make you encumbered?

That is a question to look to your university rules for. I suggest that it probably won't. That there is a clause saying that encumbered will only be enforced in cases where the debt is >£X.

My own experience with this was that in my last semester of undergrad, I accidentally had a 3 hour loan book out for several extra hours. I got a $9 fine.

So I went to the administration to pay it off. And I found out that I couldn't. They only do EFTPOS/Credit card transactions of at least $15, and they do not accept payment via cash. (These two rules came in separately several years apart and I'm not sure anyone noticed that they clash). But I also found out that encumbrance at my university doesn't occur until you have $24 or more owed.

Yes, they can do it and yes, they do actually do it.

If you look at the rules for your university it probably states something similar to this from the University of Leicester:

3.42 A student may be required to pay for certain services or facilities, including compulsory study elements of optional modules, optional residential activities, printing from computers in the open-access computer laboratories, and photocopying. Details are provided at the point of service or in departmental literature as appropriate.

3.43 No degree, diploma or certificate will be awarded to a student who has an unpaid fee, fine, or charge which is associated with the delivery of academic (rather than commercial) services. The University will not certify academic credit or previous awards for a student who has any such unpaid fee, fine, or charge. A student may make a subject access request under the Data Protection Act 1998 for data relating to his/her academic studies; such requests will be met but the data will not be presented in a certified form.

and you formally agreed to follow the rules as part of your registration. However, in 2014 the Office of Fair Trading warned universities that this may be illegal so it's not clear whether they're on legally solid ground if they do so.