octal dump of directory

Each filesystem type stores directories in a different way. There are many different filesystem types with different characteristics — good for high throughput, good for high concurrency, good for limited-memory environments, different compromises between read and write performance, between complexity and stability, etc. Your book describes a filesystem used in early Unix systems. Modern systems support many different filesystems.

The very early versions of Unix had a lot of filesystem manipulation outside the kernel. For example, mkdir and rmdir worked by editing some filesystem data structures directly. This was quickly replaced by a uniform directory access interface, the opendir/readdir/closedir family, which allowed applications to manipulate directories without having to know how they were implemented under the hood.

The reason you can't read directory contents under Linux isn't because they have to be concealed, but because features exist only if they are implemented, and this feature is pointless and has a cost. Given that the format depends on the filesystem, it's a rather pointless feature: a program can't know the format of what it's reading. It isn't a completely trivial feature to support either: some filesystems organize directories in ways that aren't just a stream of bytes, for example it may be organized as a B-tree. Some Unix variants still allow applications to read directory contents directly, for backward compatibility, but Linux doesn't have this feature (and never had as far as I can recall — it was already an obsolete feature in the early 1990s).

Yes, but:

  • modern systems store filenames differently. In original Unix, names were limited to 14 characters, with 2 bytes for inode.
  • the interface to the directory is via functions opendir, readdir, closedir rather than open, read, close to reflect the change in organization.
  • because no one has a practical need for reading 16-byte directory entries, the designers omitted the ability to read raw directory files from programs that are designed to read files.

Further reading:

  • Howto: C Programming with Directories on Linux
  • question on opendir, readdir, etc., Usenet thread in 1990 discussing the modern implementation of opendir, etc.