Why did Mathematica choose brackets for function arguments over parentheses?

The answer is quite simple. Most people want to multiply numbers without having to use the * symbol, e.g. 3x vs 3*x.

So given that this exists in Mathematica, using () for function arguments would introduce ambiguity.

Is f(x + y) meant to be f[x + y] or f*(x + y)?

This is actually a problem Wolfram|Alpha faces since it allows for all forms of inputs.

Other languages like C chose the other route, which means you must use * to indicate multiplication. Given that Mathematica's original purpose was for mathematics, I think the right choice was made.

Although Chip's answer already suffices to address the question, I would like to quote here a relevant part of the dialog by Theo Gray and Jerry Glynn in their book Exploring Mathematics with Mathematica; as there does not seem to be an easily accessible online version or preview of the book anywhere, I hope the quotation is useful:

…

Theo: Satisfied? Mathematica also knows about a whole bunch of other functions, such as trigonometric functions. One of the weird things about Mathematica that tends to annoy people for a while is that you have to use square brackets and capital letters. For example:

Sin[1.2]
   0.932039

Jerry: In other words, you're saying I can't type sin 1.2 with no parentheses, or sin(1.2), or Sin(1.2), or sin[1.2]. I must type Sin[1.2], exactly as you did. That seems like a real imposition.

Theo: Yes, you have to type Sin[1.2], exactly. There are good reasons for both requirements, and we'll see why in later chapters. If you use one of the variations you suggested above, Mathematica will warn you that you are probably making a mistake. All of your variations are legal Mathematica input, but they don't mean what you want. (For example, sin(1.2) means the variable named sin multiplied by 1.2.)

Jerry: OK, I'll live with the funny brackets for now.

…

Jerry: …now, what about square brackets? Why can't I use Sin(x) instead of Sin[x]?

Theo: Good question! There is, in fact, a good reason. Ordinary mathematical notation is inconsistent here. Round parentheses are used to mean two completely different things in traditional notation: first, order of evaluation; second, function arguments. Consider the expression k(b + c). Does this mean k times the quantity b + c, or does it mean the function k with the argument b + c? Unless you know from somewhere else that k is a function, or that k is a variable, you can't tell. It's a mistake to use the same symbols to mean these two completely different things, and Mathematica corrects this mistake by using round parentheses only for order of evaluation, and square brackets only for function arguments.

Jerry: That's a nice point. I never thought of that before. It shows how easily we adapt to nonsense. Aside from that, are you saying that mathematicians have been sloppy for centuries? That's a pretty strong statement!

Theo: Yes. Although I'm all in favor of interesting, quirky languages for writing novels and poetry (English comes to mind), it's really a bad idea to use an ambiguous language for something like mathematics. One of the great contributions of computer science to the world has been a powerful set of tools for thinking about what makes a language "good".

An alternative would be to insist on using a * for all multiplication. Then k(b + c) would always mean the function k, and if you wanted it to mean multiplication you would have to use k*(b + c). We decided it was better to remove an inconsistency than to force people to use an extra symbol. Another option would have been to have Mathematica "know" what was a variable and what was a function. This turns out to have serious consequences, and it's really not a good idea.

Jerry: Well, I didn't expect a lecture!

Theo: Sorry. Let's get back to the matter at hand. For functions, you use square brackets. Let's use the Sin function together with some round parentheses, to see how they fit:

Sin[1.2 (3 + 4)] (4 + 5)
   7.69139

Jerry: This means, Find the sine of 1.2 times 7 and multiply that answer by 9.

Theo: Yes.

…