Should I use recommendation letter from young professor I worked with or well-known professor I met in class?

Your REU evidence that X wrote you a weak letter is not very strong. I run an REU, and we accept less than 3% of our applicants. Our target participants are either legendary at small colleges (best in a decade), or excellent at top universities (top 25% this year at Harvard). Plenty of excellent people get turned away for essentially random reasons. Maybe one of your letters was submitted late. Maybe you forgot to list a course on your application that was considered essential that year. Maybe I had a bad burrito for lunch and hated your essay when I read it. Maybe there were just lots of really good applicants that year. And yes, maybe professor X (or professor Z, the other letter-writer) said or implied something negative about you.

In addition, Professor Y offering to write on your behalf after a one month course says more about her than it does about you. This does not guarantee that she will write a strong letter, and if she does she may write strong letters for a lot of people, which might be known by the people reading her letters.

Here is my advice. Go to Professor Y and tell her that you did research with Professor X and published a paper together. Then, ask her whether you should ask for X's recommendation or hers. If she really feels that you are spectacular, she will insist on writing the letter herself.

The standard wisdom about getting letters from people you worked with rather than those who merely had you in a class is only approximate, of course, and it doesn't mean that every recommendation you get from someone you worked with will be better than every recommendation from someone who you took a class with. To me, the fact that professor Y voluntarily asked if you wanted her recommendation is more important in your case. She wouldn't offer to do that unless she was confident that she could write you a good recommendation. So I think there's a decent chance that her recommendation letter actually helped you more than the one you got from professor X.