Why do toroid inductors have a square cross-section?
When making a ferrite toroid, you start with powder that has to be compressed, then sintered. It's much more practical to have a mould with substantially parallel sides, and compress in one direction. The main criterion for cross-section is manufacturability.
The theoretical shape that maximises power throughput for a toroidal transformer core is a nearly circular (sort of egg-shaped) toroid. The larger the hole in the middle compared to the cross-section, the more nearly circular it is. This shape jointly minimises the magnetic and electrical lengths, and so the losses. For any given pair of core and conductor materials, so core of ferrite, iron powder, or iron tape, and windings of Cu or Al, the optimum cross section and hole ratio will change slightly.
However, this optimum is a very weak function of the cross-sectional shape, and changes little between a circular and a squarish-with-rounded-corners area.
Space occupancy consideration
For the same volume of core material, the “important” outer dimensions of the finished product will be smaller if a square cross section is used. Hence, for example, it will occupy less floor-space on a printed circuit board.
Taking this further (see below) if a rectangular cross section is used, the outer dimension will reduce even further for the same cross sectional area and mean magnetic path length.
Inner radius saturation
Another thing to consider is the inner radius of the toroid. If the core cross section were circular then, for the same mean effective length around the core as a square section toroid, there would be a slightly shorter path that the H-field occupies on the inner radius and this would lead to a small increase in saturation at high currents.
So, what do the main toroid manufacturers do?
Maybe this can be answered by examining four popular\$^1\$ ferrite toroids from Ferroxcube. In fact these toroids are not square cross section but rectangular; the smaller dimension producing even less discrepancy between the inner and outer radius (in red is my calculated dimension): -
The same was also found for Fair-rite toroids. In other words, the preference is to have a rectangular cross section and that naturally means an even longer winding length than a square section and, an even longer winding length than a circular cross section. This is highly likely to be because it is the magnetic part of any transformer or inductor that is potentially more lossy compared to \$I^2R\$ Cu losses.
So, we usually take the mean effective path-length as that passing through the core mid-radius (i.e. where the middle of the jam is in a doughnut) and, ignore the truth that the inner radius will be slightly more prone to saturation than the outer. That’s what we do as engineers but, if we’re being picky about our design, we would need to consider it.
A circular cross section core will naturally have a smaller inner radius and, because of this, it will tend to magnetically saturate a little bit more at that inner radius (compared to a square cross section) and even more so when compared to a rectangular cross section toroid core.
Winding room
And, it follows that there is also less room to put windings in a circular section core. The smaller inner radius becomes a slightly bigger bottle-neck for the wires passing through. Going to a rectangular cross section gives even more room.
In summary
A square cross section core (for the arguments given above) will have a more uniform level of magnetic flux density from inner dimension to outer dimension and, permit more copper wires to pass through the centre. A rectangular cross section is even better.
\$^1\$ "Popular" = going to the Farnell website and choosing four different cores at the top of the list that had significant stock levels.
The total profit maximization is the wanted thing. Toroidal cores with rectagular cross-section are surely easiest to manufacture, but some rounding is useful to make winding easier for the users. The real estate cost in the finished electronic equipment is another factor which suggests rectangular cross-section. The mass of used copper when the wanted inductance and allowed resistance are fixed is an opposite factor. These all things are already said by others, but the final selection should be a result from optimization.
Nobody does it for the total process because core producers cannot fully control the users and the users must use what's available and within the budget. Some faint attempts have been seen. I swear I have seen an ad of a core seller with text "Let our engineering team to help you to select the best core for your application".