Why is this empty directory 3.5MB?

When you delete all the files from a directory, for most file systems, the directory remains the same size.

If the directory is empty,

rmdir ./directory_name; mkdir ./directory

The resulting new directory will be smaller. But as files are added it will grow larger. Do not worry about directory file size as much as the number of files in a single directory. Huge numbers of files in a single directory impact file lookup performance negatively. Even with ample inode caching.