Mathematical proofs in non-math paper: What if they are wrong?

I think you are probably right: it seems unlikely that mathematical statements and proofs included in the supplementary materials to a psychology paper would receive much scrutiny. Even if the editor took the time out to get an extra, mathematically sophisticated referee, the fact that you received no comments about this part of the paper (right?) indicates that even if "read" it was not regarded as an important factor in the evaluation of the paper.

What is the upshot of this? In a nutshell: for better and/or for worse, probably this mathematical work of yours will receive little or no attention. I don't know any mathematicians who regularly read psychology literature. Since you have just been admitted to a PhD program in mathematics, the time for people to scrutinize your undergraduate research has already mostly passed. By the time you graduate, you should/will have much more relevant material for people to focus on. It is debatable whether you should even list this paper on your CV at the end of your graduate career: I would say you probably should, but in a separate category from all of your math papers.

If I were looking to hire you for a post-PhD academic position, I honestly would not even consider reading a psychology paper you wrote as an undergraduate. I would not expect such a paper to have any substantial mathematical contributions, I would expect to be entirely unable to evaluate the significance of the paper as a work of psychology, and -- more honestly -- knowing that you put some theorems and proofs in it as an unassisted undergraduate student, I would expect this material to be a bit callow/superficial. Most undergraduates have not begun to independently engage in research-level mathematics in any way, so it is hard to hold against them whatever they put in papers.

Viewing this as a teachable moment, here are some possible takeaways:

1) I agree with Kimball and Nate Eldredge: you should do what you need to do to gather confidence in your work before you publish it. By professional mathematical standards, yes, it is a mistake to publish a piece of mathematics without soliciting the opinion of at least one mathematically qualified person. If you happen to be in a similar situation in the future, you should show your work to teachers and mentors in the mathematics department. You may have to wait what seems like a long time to get a fairly quick reading / reaction from them, but even an instantaneous reaction is valuable.

2) On the other hand, just because you're worried doesn't mean you actually did anything wrong, and it certainly doesn't mean that you did anything terribly wrong. A large percentage of papers contain minor errors. E.g. just yesterday I got an email from a collaborator informing me of a typo in a published paper (on the same subject as our collaboration, but he was not a coauthor). There was a missing "+1" from several formulas. This really doesn't distress me at all: I will make the change on the copy of the paper on my own webpage, and that's that. And by the way, I have made worse mistakes than this in published work (and in my PhD thesis): ones that do bother me a bit, but much more me than anyone else. In the realm of honest mistakes, I can't think of what you could have done in an undergraduate paper in a different field that would be so bad so as to place any clouds in the sky of your academic career.

3) Let me not completely neglect the possibility that you did something right: maybe your paper has a real mathematical contribution. If that is the case, you should not drop the matter but try to continue, improve and refine the work, with the goal of publishing it elsewhere. Again, mathematics hidden in a psychology paper is very well hidden indeed. Maybe that's not what you or the mathematical community wants.