Why is it acceptable to cite unpublished works?

You seem to be interpreting citations as a way of proving something by appeal to authority. That is not what citations are. A citation is a way of giving appropriate credit to the originator of an idea and pointing the reader to the explanation of that idea. Not giving credit to the person who gave you an important original idea is unethical, whether that idea has been published or not.

Citing a "personal communication" related to a mathematical fact (for instance) does not eliminate the need to prove that fact. Appropriate usage would be to write something like "the following was communicated to me by X. Y. [personal communication]", followed by (for instance) a theorem statement and its proof.

Note that if something is in a technical report or on arXiv, it is not unpublished. People publish blog posts, novels, and Youtube videos. Published and peer-reviewed are completely different things and it's wrong to conflate them.

What's the alternative? I can only see two:

• Don't write the text. This would also mean you have something worth sharing, but are not sharing it. This goes against the purpose of publishing in the first place.
• Write the text, but don't cite it. That's plagiarism.

Since the alternatives are undesirable, it's acceptable to cite unpublished works.

An answer by David Ketcheson highlights the importance of giving credit for ideas as a reason for citing unpublished works.

Another reason for citing unpublished works can be grey literature. The linked Wikipedia article describes the history of grey literature coming out of World War II, when the Allies produced a large amount of nuclear research that was not formally published, but still scientific important.

In my personal experience in applied ecology, grey literature can be an important source for management decisions and data. For example, a wildlife preserve manager might do controlled burns every 5 years based upon their wildlife preserve's local monitoring program. But, they would not formally publish their work as an article. Instead, they might summarize their work as part of a report that lacks peer review. However, a researcher studying fire ecology might describe the wildlife preserve's management as part of a peer review article or describe trends at the preserve.

More broadly, nuance is key to understanding what is acceptable to cite. First, different academic fields have different general expectations for how to cite non-peer reviewed science and journals usually have a distinct citation style (e.g., inline vs reference section). Second, a nuanced view is important for what is being cited. Citing facts, observations, and applied choices differ from citing conclusions.

Last, to answer your final question, what prevents authors from making bad citations of unpublished works? Nothing. But, a nuanced view of peer-reviewed articles helps here. Just because something is peer-reviewed, it is not infallible. Additionally, I've had authors make bad citations of my published works. I've joked I sometimes learn new things about my research by reading work that cites it.