What is expected in a masters thesis of a mathematics student?
I think this varies a lot. But for Germany your first question can be answers succinctly: In a Master's thesis you should show that you have potential for research.
On the other hand, expectations vary a lot between advisors. But certainly you do not have to prove a new theorem or develop a new theory.
- How much is it valued (if at all) when one applies for PhD?
I can only answer for the situation where you apply in Germany. The thesis can be a door opener if it is topic closely related to the field where you want to do a PhD. Also a good mark is important. But also in Germany hiring professors will often contact your advisors or request a reference letter and this is much more important.
- I have heard that its value is more in Europe than America, which if I were to guess I would say, may be due to no GRE like criterion. Is this true?
Not sure on this point since I can't provide a comparison with the US and also I am not sure if the situation is uniform with the EU.
From my knowledge of the US system (I did my graduate work in the US, and am currently a professor in the US), the average level of a masters thesis is relatively low. (That said, it usually does involve at least some original research).
The reason for this is the structure of Ph.D. programs in the US. Usually students are admitted to Ph.D. programs directly as undergraduates, and the first two years of the Ph.D. are similar to an MS program in Europe. Students who complete a Ph.D. don't generally write a masters thesis along the way. Rather, masters theses are usually written by students who decide in their second year not to continue with our Ph.D. program, but would still like to earn some sort of degree for their efforts. These theses are often weak (but sometimes are quite good).
Some students do use an MS as a stepping stone to Ph.D. programs elsewhere; indeed, I personally know students who successfully transferred to much stronger programs. Their MS-level work was much better than average.
In short: The degree itself won't be highly valued in the US, but doing an MS can lead to strong letters from your professors and research advisors, and these will be highly valued.
A great resource I have used to understand the quality of final thesis work for my primary focus is the Open Access Theses and Dissertations which has thousands of master's and Ph. D. final publications. Research this website using your topic and you will see what amount of research is involved, differences and similarities between schools, methodologies, etc.
In addition, a great site for further publications is http://Arxiv.org . Many thesis in the U.S. are 'sandwich' publications, involving an assortment of publications published while student is performing research.