how to prevent a user from changing his home directory permissions?

One way is to use per-user groups (i.e. one group for each user) and then set the home directory permissions to root:smith, mode 0770.

Another (more hacky) way is to script this: Create a script that inspects all home directories (get them via setpwent()/getpwent()) that reside under /home (e.g. not /root) and make it either warn when there's a discrepancy or change the permissions on the spot.

I've done the latter myself in a multi-user environment in the past and worked for years like a charm.

Your wish is only possible if the OS and the filesystem in question support the ACL standard that is used by NTFS and ZFS and that is standardized by NFSv4.

So whether you are able to do what you like depends on OS and filesystem.

Traditional UNIX rules are that user ownership of a directory also permits access mode changes. With ACLs, you are able to deny the user to change ACLs on his own directory.