Why do German universities rank so low internationally?
There are lots of biases in international academic rankings, they tend to favour US and UK universities because they use criteria which are mostly relevant for a market-oriented academic system, such as the US/UK ones.
In many European countries the academic system is not market-oriented, or at least not as much as in the US and the UK: students fees are very low because higher education is mostly funded by public money. Universities are seen as a public service offering education to society as a whole, rather than a kind of commercial institution which sells knowledge and qualifications to those who can afford it. As a consequence Non-US/UK universities are not as incentivized to attract international students, they don't make big efforts to play the competition game since their income doesn't depend (at least not much) on their international ranking.
Of course this is only a simplified explanation, but this is the main reason why US/UK universities perform better than German universities (among others) in international rankings.
[Edit] Originally this was only a quick answer to explain what (I think) is an important structural and cultural difference with respect to international rankings between UK/US-like academic systems and others. My point was only to emphasize this difference to the attention of prospective students looking at these rankings, knowing that this is a common source of misunderstanding among them. I didn't imagine that the question and my answer would attract so much attention. I gladly admit that this is a rather simplistic answer, and I'm happy to see that other answers have done a much better job than me at analyzing other aspects of the question in detail.
Whence university rankings?
University-wise reputations are somewhat of a self-amplifying phenomenon:
University-wise reputation is an established thing. This is opposed to the reputation of a department or faculty.
Students and academics want to be at a well-reputed university. Thus, well-reputed universities can be more selective and get better students and staff.
Better students and academics at reputed universities raise the level of teaching and research through existing qualities, higher teaching levels, cross-fertilisation, researchers having access to better students, etc.
Reputed universities actually are better as they produce better graduates and research, attract more funding, etc.
University-wise reputation becomes a reasonable criterion and gets further established. Go to Step 1.
However, without such a process, university-wise reputations hardly makes any sense: Most interactions in universities happen within departments or at least within faculties, and thus there is no reason to assume the qualities of different faculties correlate with each other. For example, the idea that a university with an excellent science faculty also must have a good law faculty is absurd (except for the above mechanism).
German specialities
In some countries, such a process has happened; in Germany, it mostly didn’t. Germans rarely think about the reputation of a university, but rather about the reputation of a department, and there are indeed strong variabilities between those. This is at least partially due to historical factors, but there also some systemic factors:
In the German education system, students specialise on a field upon entering university. This reduces the interactions between departments as compared to other systems.
German universities are primarily funded by taxes not by fees and donations. People do not make donations to their alma mater; they pay taxes. Reputed universities cannot raise higher fees. There is no rich-get-richer amplification furthering the above process.
The German culture is rather egalitarian and particularly holds to the ideal of providing free and equal education to everybody (or in case of universities, everybody with certain prerequisites). The concept of an elite school or university is not generally well regarded. (Mind that whether those egalitarian ideals are actually achieved is another question.)
A considerable amount of research in Germany happens at dedicated research institutes (mostly Max Planck, Helmholtz, Leibniz, and Fraunhofer Institutes) that are usually only loosely affiliated with universities. Whatever reputation these institutes acquire does not fully rub off on universities (in public perception as well as methodic rankings).
Note how the German universities that have a somewhat generally good reputation tend to be located in beautiful and expensive cities and thus are more attractive to students with rich parents (which despite all egalitarian tendencies have better prerequisites). Finally, mind that there is a (disputed) initiative that may initialise the above process, but even it does, I would expect it to take decades to show effect – in particular if you measure success in Nobel prizes (see below).
Consequences
Now, all of this leads to German universities scoring badly in different ways:
The effect of a few good departments at a university gets lost in averaging. The good departments in Germany are simply not clustered at a single university.
Scoring high in a university ranking is not such a relevant factor for universities, and thus they have no incentive to game those systems.
Even top departments are not that attractive to students that they can be as selective as a globally high-ranking university. Students simply select their place of study by other criteria.
A considerable portion of research successes happens at research institutes and not at universities and thus does not boost any university’s ranking.
For ARWU (Shanghai) in particular: This ranking mainly counts the extremes (Nobel prizes, high-impact papers, etc.). Broadly speaking, it looks at the highest percentile of research happening, not the median, average, or similar. Most of the aforementioned points are particularly bad for achieving such extremes, while hardly affecting the average quality of research and teaching.
A significant portion of this phenomena may simply be due to core problems in the methodology of these rankings systems. For example:
- ARWU rankings are based 50% on sparse outlier information for "glamour" rather than "normal" science: 30% for Nobel and Fields winners (which instantly excludes most research fields) and 20% for Nature & Science papers (which form a quite small fraction of even most famous researchers' significant output). Here, the top two German universities are ranks 51 and 54.
- QS World University rankings are based 50% on reputation surveys, which will tend to make "top ranking" a self-reinforcing phenomenon. Here, the top two German universities are ranks 50 and 63.
- Times Higher Education rankings are a bit more egalitarian, with 35% based on reputation surveys. Here, the top two German universities are ranks 32, and 41.
- US News rankings are also less reputation-based, with 25% based on reputation surveys. Here, the top two German universities are ranks 43 and 56.
I find it interesting and possibly significant that the German universities rank higher in the less outlier-based and reputation-based ranking systems. Remember also that top 100 is still quite high, given that there are approximately 1000 graduate schools and thousands more 4-year colleges in the USA alone.
Furthermore, even the "objective" measures like "citations per faculty" and "number of papers in top 1% of field" are going to be dominated by the high tail of the distribution, rather than the actual educational and research opportunities provided to the median undergraduate or graduate student.
Thus, if the German universities have not been optimizing for these metrics, as suggested by other answers, I would expect the best ones to show up only in the medium-high rankings, even if they are extremely good institutions (as they have been).