Android - Why can apps access the whole SD card?

SD cards generally use the FAT file system, which does not allow for filesystem permissions. This means that the Android OS would have to manage a custom list of files, their permissions, a mapping of apps to files/folders, etc. in order to prevent apps from accessing the whole card. This would be a terrible mess. Not only complicated, but it would require a lot of storage space. And then one day you take out your SD card and remove some pictures, and the entire system breaks. You get the idea.