why is there dark fiber?

Dark fiber (unused fiberoptic cables) is a confusing term for me. Why would any company build so much excess capacity?

The marginal cost of adding one more fiber when digging up a street is very, very small relative to the cost of digging up the street in the first place. So when you decide to dig a trench across the country, string a fiber across the ocean, etc you typically put more than one in place so that in the future you don't have to do it all over again when you need more capacity.

In addition, much of the cost of the fiber link is actually the equipment at the ends of links (WDMs, receivers, pump lasers for amplifiers, etc). Burying a lot of dark fiber therefore allows you the option of paying to install more lines sometime later in the future (by buying more receivers) for a relatively small up front cost (just whatever it costs to put one more fiber in the trench).

In addition to the previous answer just a comment on terminology, If industry speaks about Dark Fibre, especially renting them, this typically means you buy the fibre capacity and have to take yourself care for transmitting/receiving and multiplexing.

It does not mean the Connection is not used or in excess, it means you are not bound to the provider equippment. Of course before you can rent those lines they do have to exist and be unused. The lines are digged into the ground in expectation of beeing sold (managed or unmanaged).