How to set ulimits for mongod?

Modify /etc/security/limits.conf with what you need. Example:

user soft nproc 64000

This line will set the number of processors (-u) to 64000 for "user". Soft/hard limits can be the same (soft allows spikes while hard prevents spawning).

The newer version of the CentOS mongod startup script (/etc/init.d/mongod) has the default settings built into the start option:

start()
{
  # Recommended ulimit values for mongod or mongos
  # See http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings
  #
  ulimit -f unlimited
  ulimit -t unlimited
  ulimit -v unlimited
  ulimit -n 64000
  ulimit -m unlimited
  ulimit -u 32000

  echo -n $"Starting mongod: "
  daemon --user "$MONGO_USER" "$NUMACTL $mongod $OPTIONS >/dev/null 2>&1"
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/mongod
}

If you want to change the values and not modify the script, copy the script to /etc/init.d/mongod-custom, then edit the custom init script, set your values and change chkconfig to use mongod-custom instead of mongod.

To add to thaspius's answer, the limits seem to be set as he describes in the init script, but I was still getting the warning

[initandlisten] ** WARNING: soft rlimits too low. rlimits set to 1024 processes, 64000 files. Number of processes should be at least 32000 : 0.5 times number of files.

This seems to suggest that files limit defined in the init script had taken effect, but not the processes limit. Also following Nathan C's answer and adding

mongod soft nproc 64000

to /etc/security/limits.d/90-nproc.conf and restarting the system solved the issue.

If anyone is able to shed light on why that was necessary despite having the values in the mongodb init script I'm all ears!