Why do later versions of Windows continue to use shortcut files instead of symbolic links?

A number of reasons, I guess

1. You can store different levels of compatibility against several different shortcuts to the same EXE as they're interpreted by the shell, rather than the file system.
2. Certain shortcut links don't actually exist on the file system. Some of them are simply references to GUIDs, or special strings interpreted by the shell.
3. You can't include switches in a symlink. You can point to the EXE, sure, but you can't tell that EXE any further arguments.
4. You can't choose an icon for a symlink.
5. You can't choose what directory to work from in a symlink.
6. Shortcut files don't just have to point to files, they can be hyperlinks or protocol links (In the case of a .URL file).
7. LNK files can exist on any file system. Symlinks are handled by the file system itself, in the case of Windows, NTFS.
8. There's no real need to replace them. They work, they're tiny, they can be scaled up in the future should there ever be a need for more functionality to be added to them than listed above.
9. Administrative rights are required to create a symlink (For good reason - otherwise redirection of innocent files to malicious ones can be executed with very little work)

There will be more reasons than this, but I think that's enough to get you started :) - There's a link provided by @grawity here that will give some further reading on parts of this topic.

A symlink is nothing more than a path wrapped up in a very small amount of filesystem magic. There are any number of ways it can become invalid ("broken"), most of which involve one or more files or directories getting renamed. Since Windows is consumer software, you may have a large number of very poorly designed programs running on a "typical" installation. As a result, this kind of breakage is a lot harder to avoid than on a server where (in theory) every program that touches the disk is a known quantity.

Shortcuts are immune to most forms of breakage since they track their targets independently of path. This makes them more user-friendly. They are specifically designed for consumers, with a "just do what I mean and don't bother me about the details" approach.

Now, you could use hard links for that (to some extent), but hard links have a number of complicated properties which make them unsuitable for consumer use. In particular, files get new inode numbers entirely too easily and some backup software breaks rather spectacularly when confronted with hard links. The former could (perhaps) be solved with filesystem tunneling (which is in fact how shortcuts solve a related problem), but the latter is a much harder problem.

(I should probably also note that "solving" hard links with tunneling is decidedly nontrivial since it's not just a matter of reattaching metadata that's "gotten lost." Inodes are bound up in the disk allocation scheme, so you can't just arbitrarily merge or reassign them after the fact without a fair bit of legwork. Since shortcuts use other metadata that can be easily tunneled, like the creation time, they don't have this issue.)