My paper was retracted – how can I handle this in my future career?

In the following, I may misinterpret what happened. In this case, please correct me. I just don’t want to append “if I understand correctly” to every second sentence.

Let’s first consider whether you did anything wrong: As your “collaborator” designed the research underlying your paper, he was entitled to authorship. You understood that he waived that right, which may be correct or at worst an honest misunderstanding. However, when submitting the paper, you did not disclose this situation to the journal, and as a result, they could not double-check that your collaborator waived the authorship, etc. Thus I can understand the journal’s reaction. This failure of disclosure is the one thing others can truly hold against you, but I would not consider this to have career-ending severity, in particular given your collaborator’s abusive behaviour.

Second, how bad is the retraction note? If I were to read that note without knowing anything else about it, I would primarily be confused, asking myself things like: “How is overlapping and being based on unpublished research a bad thing? Why was that research not published?” It’s nothing which clearly says that you committed misconduct. In particular, I have no idea what actually happened and would not judge you without hearing your version of the story. Of course, others may jump to wrong conclusions, but then you probably do not want to work with such people anyway.

Now, how can you handle this situation in the future? I strongly suggest to be upfront about it. Should you not mention it and somebody will find it, this can get you into big trouble. Keep in mind that as long as your collaborator is around, they may inform any of your employers about it. Also, sitting on this ticking bomb will not be good for your psyche. For example, mention the paper in some special section on your CV, noting that it was retracted due to “misunderstandings/disagreements about waived authorship” (be careful not to present it as an achievement). This is accurate and explains the retraction note. Hiring committees and similar may ask you about it, but then you can tell the entire story, presenting your evidence if applicable. When you do so, make clear that you are aware of your mistakes, but also that your collaborator was far worse.

Finally, follow the journal’s recommendation and contact your collaborator’s university about this. If your abusive collaborator was officially discredited, this could substantially strengthen your point. Whether this is feasible depends on how strong your evidence is, and I strongly recommend that you seek professional help with this (your current university may provide some).

Wrzlpmft addressed most of the questions. I would like to add an additional answer to this part of the question.

How can I handle it in future academic applications.

You should ask the journal to change the retraction notice to indicate your agreement.

The prospective supervisor committed misconduct. You were correct to stop working with him. However, he does not seem to have waived his coauthorship. Unfortunately, his misconduct does not remove his coauthorship. So the professional thing would be to agree that the paper should be retracted so as to demonstrate that you respect his coauthorship. This does not imply that you respect him as a person. Anyone evaluating academic applications will not select someone who does not respect authorship.