Why is the size of a directory always 4096 bytes in unix?

It's the initial size necessary to store the meta-data about files contained in that directory (including names). The initial allocation equals the size of one sector, but can grow above that if necessary. Once allocated, space is not freed if files are removed, to reduce fragmentation.

For example:

$ mkdir testdir
$ cd testdir
$ ls -ld .
drwxr-xr-x 2 matthew matthew 4096 2007-12-03 20:28 ./
$ for ((i=0; i<1000; i++)); do touch some_longish_file_name_$i; done
$ ls -ld .
drwxr-xr-x 2 matthew matthew 36864 2007-12-03 20:29 ./
$ rm some_longish_file_name_*
$ ls -ld .
drwxr-xr-x 2 matthew matthew 36864 2007-12-03 20:29 ./
$ cd ..
$ ls -ld testdir
drwxr-xr-x 2 matthew matthew 36864 2007-12-03 20:29 testdir/
$ rmdir testdir ; mkdir testdir
$ ls -ld testdir
drwxr-xr-x 2 matthew matthew 4096 2007-12-03 20:29 testdir/

source

Sometimes 4096 bytes is the smallest allocation unit for some filesystems. That's why directory has 4096.

The same thing apply to files. Even though some files might report fewer than 4096, they are actually taking al least 4096 of storage from disk.

4096 is reserved to reduce fragmentation, because often the actual size of the metadata contained will fluctuate based on the directory contents. If it is constantly growing and shrinking (say it contained log files or dynamic content) over time it could hurt performance. This likely wouldn't happen with one folder, but across the whole file system it would add up quickly.