When should one recommend rejection of a manuscript versus major revisions?

Many of the issues you raise are not relevant. Who wrote it, for example. Likewise scooping isn't relevant, though you may need to withdraw due to a conflict of interest. If the research is really parallel, not plagiarized, then scooping is fine.

But for the main question, ask yourself, what would be the quality and impact of the paper if the errors are (mostly) corrected? Is it innovative? Are the main results important? Is it "interesting". If the answer is no, then the paper should be rejected. Otherwise suggest major edits and let the authors see what they can do.

But I think the conflict of interest issue is especially important in your case. If it is eventually rejected and it is learned that you were a reviewer but later published something similar it won't help your career. You will be open to a plagiarism charge whether warranted or not.

If you think the issues with the manuscript can be fixed, the authors should be given the chance to do a revision - if those are major issues, or a large number of issues, then a major revision.

If you think the issues cannot be fixed (or after fixing it would be in essence a new manuscript), then the manuscript should be rejected.

If there were an objective way to answer this, peer review would not be so random. Unfortunately there isn't. Still, the standard guideline is that if the issues are fixable, recommend revision; if the paper is fatally flawed, recommend rejection. Alternatively, if the issues are (relatively easily) fixable, recommend revision; if they will take a lot of effort, recommend rejection.

Examples of issues to recommend revision:

• Poor (but still comprehensible) English.
• Paper does not cite X, Y, Z, but it should.
• Paper does not clearly define [term] even though it uses [term] extensively.
• Paper does not perform [cheap experiment] to verify [claim], but [claim] is plausible - recommend they do the experiment instead.
• Paper has a curious feature in the data that could refute the paper's thesis, and the authors do not treat it - recommend they examine the feature instead.

Examples of issues to recommend rejection:

• Incomprehensible English.
• Paper is provably wrong.
• Paper is provably inferior, e.g. a paper proposes a new method to do X with 80% effectiveness, but another known method is already capable of doing X with 85% effectiveness.
• Paper is plagiarized / violates academic ethics (e.g. it is submitted without the consent of all authors).
• The paper, even if correct, is not interesting enough for the journal (or not within its scope).
• Paper's central thesis is [claim], which is not supported by the analysis conducted. For example, if the paper says people prefer X to Y, but the sample of people it used is clearly biased, and there is no analysis of how the sampling bias might affect the results.
• Paper lacks basic features, e.g. it analyzes the effect of X on Y, but doesn't have a control sample even though they really should have designed the experiment with one.

All that said, not everyone will agree with all of the above, e.g. some might recommend the authors repeat an experiment with the control. So you'll have to exercise judgement.

Based on your description of the paper you are reviewing, it reads to me like it's more reject than revise. The most problematic points are points #2, #3, #4 and the last one. A paper that contradicts itself is a clear problem that should never have arrived at you in the first place (assuming you didn't misunderstand). Scooping is not so much a problem as it is the fact that the data was thrown together without critical review. If as you describe it the data is nonsense, then the results are in jeopardy and could be completely wrong.

If you are still concerned, you could recommend revision and give the authors a chance to improve their work. If the revised manuscript is still terrible, then recommend rejection.