What roles do DAC (file permissions), ACL and MAC (SELinux) play in Linux file security?
When a process performs an operation to a file, the Linux kernel performs the check in the following order:
Discretionary Access Control (DAC) or user dictated access control. This includes both classic UNIX style permission checks and POSIX Access Control Lists (ACL). Classical UNIX checks compare the current process UID and GID versus the UID and GID of the file being accessed with regards to which modes have been set (Read/Write/eXecute). Access Control List extends classic UNIX checks to allow more options regarding permission control.
Mandatory Access Control (MAC) or policy based access control. This is implemented using Linux Security Modules (LSM) which are not real modules anymore (they used to be but it was dropped). They enable additionnal checks based on other models than the classical UNIX style security checks. All of those models are based on a policy describing what kind of opeartions are allowed for which process in which context.
Here is an example for inodes access (which includes file access) to back my answer with links to an online Linux Cross Reference. The "function_name (filename:line)" given are for the 3.14 version of the Linux kernel.
The function inode_permission (fs/namei.c:449) first checks for read permission on the filesystem itself (sb_permission in fs/namei.c:425), then calls __inode_permission (fs/namei.c:394) to check for read/write/execute permissions and POSIX ACL on an inode in do_inode_permission (fs/namei.c:368) (DAC) and then LSM-related permissions (MAC) in security_inode_permission (security/security.c:550).
There was only one exception to this order (DAC then MAC): it was for the mmap checks. But this has been fixed in the 3.15 version of the Linux kernel (relevant commit).
DAC == Discretionary Access Control, http://en.wikipedia.org/wiki/Discretionary_access_control
MAC == Mandatory Access Control, http://en.wikipedia.org/wiki/Mandatory_access_control
ACL == Access Control List, http://en.wikipedia.org/wiki/Access_control_list
The ACL specifies the controls to be applied by the method of control, DAC or MAC. MAC is explicit, centrally controlled, and does not allow users to grant authority to an object unless they have explicit permissions to do so, whereas DAC allows users to grant other users access to objects they can access.
MAC ACLs will always be applied to a request first, and if access is denied processing stops. If access is permitted then the DAC ACLs are applied, and again if access is denied processing stops. Only if access is granted by both MAC and DAC ACLs can the user access the object they requested.
SELinux is a MAC implementation for Linux (there are others), while the traditional rwx file permissions, combined with the owning user and group form the complete DAC ACL. The SELinux 'policy' is essentially the MAC ACL.
Sorry to quibble, but I think some of the answers here might be incorrect. Directly from Fedora's http://docs.fedoraproject.org/en-US/Fedora/13/html/Security-Enhanced_Linux/sect-Security-Enhanced_Linux-Working_with_SELinux-SELinux_Contexts_Labeling_Files.html:
SELinux policy rules are checked after DAC rules. SELinux policy rules are not used if DAC rules deny access first.