linux limits.conf not working?

limits.conf is read by pam_limits.so, so you need to check that this module is enabled in /etc/pam.d/su. On my system, this was commented out by default:

# Sets up user limits, please uncomment and read /etc/security/limits.conf
# to enable this functionality.
# (Replaces the use of /etc/limits in old login)
# session    required   pam_limits.so

Once uncommented, the module will start working immediately. If you still have problems, check your auth log (maybe /var/log/auth.log), which is where limits-related errors will be written.