Why do papers not have a table of contents?

The purpose of an overview paragraph and a table of contents are different:

• The table of contents serves to quickly find a specific section of a document so you can start reading there or extract a specific information. This is something that rarely makes sense with most papers, except if you start at the conclusion/discussion, which is however easy to find anyway. Even if you want to do this, finding the table of contents would cost you more time than it saves you, given the shortness of most papers. So my answer to your titular question would be: Because there is no need for it.

  The main exception to this are long review papers, which however usually have a table of contents.

• The overview paragraph serves to briefly summarise the paper and explain its semantic structure to the reader, usually somebody who is reading the paper from back to front. This serves to address certain expectations the reader may have had after the introduction (“don’t worry, we didn’t forget this important aspect”) and draw connections between sections (“our work in Section 3 provides us with a framework to evaluate our method in Section 4”). All of this is not possible in a table of contents.

  Now, some papers just have a straightforward structure, and there is nothing about it that needs to be elaborated to the reader. In this case an overview paragraph would indeed be nothing but a written-out, tedious-to-read table of contents and should be skipped in my opinion.

In fact, the idea that a math paper that is not short would be well served by having a table of contents (TOC) is one that I "discovered" at some point relatively recently. Based on a perusal of my own recent papers and preprints here, it seems that the threshold for a TOC being useful in a math paper is somewhere between 13 and 17 pages. (I believe that I put a TOC in a six page paper a couple of years ago and took it out in response to editorial comments that it was not necessary.)

Let me make some further comments:

1) Some structural redundancy in a paper is a very good idea. Often the introduction repeats things that were just said in the abstract and these same things occur later in the paper as well. Although there is a lot of room for good writing in making this repetition most useful for the reader, certainly such repetition is highly useful if done properly. I don't think having a TOC obviates having a part of the introduction in which you say what you will do in your paper. Rather, having a TOC frees you up from saying what will be in your paper in almost exactly the same comprehensive way as a TOC in paragraph form. Instead, in the introduction you can talk about why you are including the material you're including in each section. See e.g. here for a paper with a TOC in which the introduction nevertheless ends roughly as you say is not necessary. Here I think both are helpful because they serve separate functions: if you want to know why we are including the Schwartz-Zippel Lemma in our paper, the introduction explains it. If you want to know exactly where each of several versions of this result occurs, you should see the TOC.

2) I have found that a TOC is perhaps even more helpful for the authors than for the readers, especially when there is more than one author. In a medium to long math paper, sometimes you get the idea of (e.g.!) taking subsection 2 of section 5 and inserting it in between subsections 3 and 4 of section 2 because you see something to be gained by such an adjustment of the logical sequence. But then you also have to worry about what else might be disturbed by such a change. It is then really helpful to have a detailed TOC, because rather than having to keep a mental picture of the intermediate pages of your paper, you can flip to the TOC and allow your eye to do some of the work. If you just wrote all the material yourself, maybe it is not so bad to keep it all in mind. But if some of it was written by your coauthors, this gets hard. Moreover, if you do this kind of thing more than once, then your encyclopedic mental picture of the paper does not update so easily.

3) A TOC is also more or less useful according to the titles of the various subsections of your paper. If you have no titles at all (which is not unheard of), a TOC would be ridiculous. If the title of Section N is "the proof of Theorem N," then a TOC does not seem necessary (and good for you for implementing such a simple, transparent format for your paper).

Of my limited experience with older papers I am inclined to conclude that papers used to be much shorter (most papers before 1980 I read were about 2 to 5 pages). At that length an additional table of contents was not necessary (in addition to the one of the journal issue - at that times an issue was an actual booklet and also would have needed additional paper to print and ship). So I suspect that there is a historic reason.

Note that some journals which usually publish long papers have a toc nowadays (e.g. Acta Numerica).