What prevents reviewers from providing biased and very negative reviews?
I think the main factor that makes the peer-review system to not fall in the trap you described is that academics are finally interested in the advance of science. If I have to review a good paper, then I enjoy it and want to see it published. If I review an almost good paper, I suggest some improvements and enjoy if these are incorporated and the work is published. The very mild advantage I may have if I reject more papers does not really seem to be worth it, if I have to act unfair and unreasonable to get this advantage.
Another factor may be: The reviewer is not anonymous for the editor. Editors are often influential people and writing consistently bad and unfair reviews will make you look like a mean and unfair person. Also, the editor will not choose you again as a reviewer if you write such reviews and you will be effectively eliminating yourself from the review system for that journal (while you can still submit papers).
In view of the bounty I gave the question another thought and here is one more thing: Community. Science is well organized in communities (e.g. I consider myself as part of the mathematical community, the applied math community, the community on mathematical imaging, the optimization community and some more). As part of a community one has a sense that there are some rules one should follow to be a valuable member of the community and of these rules is fairness. Being unfair feels like a misconduct by the unwritten community rules. So even though one may get along with several cases of unfair behavior, it feels like one is cheating the system. But staying a respected member of the communities is very important both for the scientific standing and also for productivity. This reasoning also explains that bad behavior sometimes happens, when several competing communities form which are somehow "enemies of each other": One can stay a respected member of one of the communities while still treating members from the other community unfair.
The term competitors does not really describe the relationship between different scientists in the same field. Of course there is some competition for grants or such, or even to put one's name in a new result, but some aspects are deeply different:
A publication in your field is a good thing, even if it's not from yourself. It makes your area important, and alive. It's actually very important when it comes to applying for grants.
You cannot benefit from someone else's work by writing negative reviews. There is no way to simply slow down publication of one paper in order to publish the result first.
A rejected paper will not slow down the "competitors" research. Sure they will have to improve it and submit it elsewhere, but this takes a marginal time compared to the research process itself.
Dishonest reviews are clearly dangerous, as some people will see your review with your name on it (I'm thinking of other reviewers on such systems as Easychair). These people might very well be the ones reviewing your grant application a few weeks later.
why would one risk being exposed as dishonest, when anyway there are several reviews for the same paper, and any deep difference should trigger an in-depth investigation by the editor?
The most frequent bias I have seen are review of the form "you should cite those 5 obscure and vaguely related papers all from the same author", which clearly indicate reviewers in need of citation, but nothing much worse.
I have nothing to do with academy, so take my answer just as something that I'd expect based on my understanding of human interaction and behaviour.
Reputation
The main resource you have is your own reputation, and that will be severely harmed if people are accusing you of being unfair, biased or "not objective enough". You might be able to get away with it once in a while, but overall, it's quite enough for the system to work. Worst case scenario, you slow down the propagation of the work in question, but in practice, the paper will find another way (another reviewer, journal...). Even if you succeed, you risk harming your reputation, which is extremely important in a field centered around collaboration with peers and promising understudies.
Competition
You only considered competition between individual scientists, which is mostly a thing of (1) competing for grants and (2) competing for reputation. I've already dealt with reputation. Competition for grants might be important for you if you're trying to adapt the process for something like performance reviews - it's the clearest cut case where hurting others can help you. However, it doesn't have much to do with the peer review process - that would indeed introduce a very strong motivation to be "as unfair as you can be without actually appearing unfair".
However, there's also another competition going on - that between individual reviewers and their journals. If your paper was rejected based on grounds that are seen as fair and objective, you'll likely also be rejected by other journals. If not, the other journals might jump on the opportunity to publish your paper, while also implying that another journal has treated the paper unfairly. You can't do this very often if you want your journal/reviewers to keep being relevant!
Points to take away
If you want to use a similar system for another domain, make sure that similar incentives are at play.
- Have multiple independent reviewers, and let people choose their reviewer (while the reviewer has a chance to decline).
- Make sure there's not a lot of "authority" in play - for example, superior-underling relation doesn't make for good peers. Peer review works best with consensus and with reasonably objective / shared values.
- Make everything public (in the team / company). No anonymity, no "hidden" reviews. This is necessary for the reputation-based controls to work. In a way, it's a redundancy in the peer review system - it allows people to "review" the reviews themselves.
It works best in mostly flat hierarchies. Thinking in terms of a performance review in a company, peer review will be a poor choice if managers order people around. On the other hand, if managers have to persuade others to follow with their plan, it might work great :)