what is OS hardening in linux or unix?

To quote wikipedia:

In computing, hardening is usually the process of securing a system by reducing its surface of vulnerability. A system has a larger vulnerability surface the more that it does; in principle a single-function system is more secure than a multipurpose one. Reducing available vectors of attack typically includes the removal of unnecessary software, unnecessary usernames or logins and the disabling or removal of unnecessary services.

There are usually guides by the OS vendor. Look for example at the Debian guide: http://www.debian.org/doc/manuals/securing-debian-howto/

Hardening is a process of securely configuring weak(vulnerability) point of a system like there may be unused port, services or useless software running that may create weak point in your system. these weak point may be used by other's to enter in your system.

For example in Linux systems there is a file called /etc/hosts is used to allow/disallow users as per policy. So you may need to change the permissions of this file chmod 700 /etc/hosts.allow or chmod 000 /etc/hosts.allow

For OS(Linux, Unix) hardening go to http://www.itsecurity.com/features/ubuntu-secure-install-resource/ http://www.security.berkeley.edu

You can use tool like PSysHard Hardening Framew0rk