Why is su world executable?

One point that is missing from ilkkachu's answer is that elevating to root is only one specific use for su. The general purpose of su is to open a new shell under another user's login account. That other user could be root (and perhaps most often is), but su can be used to assume any identity the local system can authenticate.

For example, if I'm logged in as user jim, and I want to investigate a problem that mike has reported, but which I am unable to reproduce, I might try logging in as mike, and running the command that is giving him trouble.

13:27:20 /home/jim> su -l mike
Password:(I type mike's password (or have him type it) and press Enter)
13:27:22 /home/mike> id
uid=1004(mike) gid=1004(mike) groups=1004(mike)
13:27:25 /home/mike> exit  # this leaves mike's login shell and returns to jim's
13:27:29 /home/jim> id
uid=1001(jim) gid=1001(jim) groups=1001(jim),0(wheel),5(operator),14(ftp),920(vboxusers)

Using the -l option of su causes it to simulate a full login (per the man page).

The above requires knowledge of mike's password, however. If I have sudo access, I can log in as mike even without his password.

13:27:37 /home/jim> sudo su -l mike
Password:(I type my own password, because this is sudo asking)
13:27:41 /home/mike>

In summary, the reason the permissions on the su executable are as you show, is because su is a general-purpose tool that is available to all users on the system.

Historically (on non-GNU unices), it wasn't, or at least it manually checked if you were in a group called "wheel" permitted to su. The GNU version of su did not reproduce this functionality because of RMS's ideology about access control at the time:

Why GNU `su' does not support the `wheel' group
===============================================

   (This section is by Richard Stallman.)

   Sometimes a few of the users try to hold total power over all the
rest.  For example, in 1984, a few users at the MIT AI lab decided to
seize power by changing the operator password on the Twenex system and
keeping it secret from everyone else.  (I was able to thwart this coup
and give power back to the users by patching the kernel, but I wouldn't
know how to do that in Unix.)

   However, occasionally the rulers do tell someone.  Under the usual
`su' mechanism, once someone learns the root password who sympathizes
with the ordinary users, he or she can tell the rest.  The "wheel
group" feature would make this impossible, and thus cement the power of
the rulers.

   I'm on the side of the masses, not that of the rulers.  If you are
used to supporting the bosses and sysadmins in whatever they do, you
might find this idea strange at first.

You can find a lot more on the matter by Googling "wheel group rms" or similar.

However, since the permissions on su are -rwsr-xr-x there's nothing stopping them from attempting to brute force the root password.

Yes. Assuming your usual Linux system, the pam_unix.so module does delay a failed authentication attempt by ~two seconds, but I don't think there's anything to stop simultaneous attempts.

Failed attempts are logged of course:

Aug 16 22:52:33 somehost su[17387]: FAILED su for root by ilkkachu

Brute-forcing a password should show up prominently in the logs, if you have any kind of system for monitoring them. And if you have untrusted local users, maybe you should. Untrusted local users could also attempt to use any local-only privilege escalation exploits, and they're much more common than remote privilege escalation.

What's further puzzling to me in that it doesn't even seem useful.

Of course it's useful, it allows you to elevate yourself to root, if you know the password.

It allows me to run su directly instead of needing to do sudo su, but this is hardly an inconvenience.

sudo su is somewhat redundant. There's no need to use two programs that are meant to allow you to run programs as another user, one is quite enough. Just use sudo -i or sudo -s (or sudo /bin/bash etc.) if you want to run shell.

Am I overlooking something? Is the world execute permission on su just there for historic reasons?

See above. Well, not all systems have sudo as an alternative, I'm not sure if you consider that historic.

Are there any downsides to removing that permission that I haven't encountered yet?

Not really, no as far as I know. I think some systems have su set so that only members of a particular group ("wheel") can run it. You can do the same to sudo if you like (chown root.sudousers /usr/bin/sudo && chmod 4710 /usr/bin/sudo).

Or you could just remove either or both of the programs if you don't need them. Though on Debian, su comes with the login package, so it's probably not a good idea to remove the package as a whole. (And pairing login and su together like that does seem somewhat historic.)