What are the criteria to choose a secure operating system?

There are many criteria you can use for choosing a secure operating system. This includes everything from the features explicitly supported to the overall goal of the system itself. Although this answer turned out rather long, it actually only touches on a small subset of considerations.

If this is tl;dr and you want a quick summary with recommendations, skip to the very last section. Otherwise the following will give you a basic understanding of how different operating systems differ in their security. It is by no means exhaustive, but should fit for such a broad question.

Userspace hardening

A hardened userspace is important in threat models where your adversary is able to send attacker-controlled data to your system where it will be parsed by potentially insecure code. This can manifest itself as anything from media files to the HTML and CSS you view on your web browser. You need to assess whether you will be exposed to such data, and under what circumstances. Is your adversary able to get you to visit a malicious website? Can they get you to view a malicious video file or image file? Can they communicate with any remote services on your computer? Etc.

An operating system that is hardened makes use of built-in security features to reduce the probability of success or overall impact of exploits. An operating system that provides you with pre-compiled software (a binary distribution, in *nix terminology) must apply this hardening at compile time. The hardening typically involves instrumenting executable code to reduce the amount of information an attacker has about the memory layout, or to complicate common exploits.

• ASLR (Address Space Layout Randomization) is a ubiquitous security feature in virtually all modern operating systems. In general, it functions to randomize memory layout, forcing an attacker to additionally exploit an information leak before they are able to exploit any vulnerabilities that require knowledge of memory layout. Different kinds of ASLR randomize different aspects of memory. The more complete the ASLR is, the higher the security. ASLR is provided by the kernel of the operating system and applies to all executables that are loaded. ASLR first appeared in PaX, a component of the grsecurity hardening patch.

• PIE (Position-Independent Executable) is a compile-time feature that allows an application to execute a function without knowing its absolute address. If PIE is not supported, ASLR will be significantly more limited in its capabilities, effectively allowing only randomization of the base. When PIE is supported, even individual libraries can be placed at random offsets. Because PIE reduces performance slightly, some operating systems only compile sensitive applications this way. Debian is an example of this, with very limited PIE support. Fedora on the other hand compiles everything with PIE to maximize security.

• SSP (Stack Smashing Protector) is another compile-time feature that adds checks before function returns that checks for stack overflows. It can be implemented in several ways, but one of the most common is the stack canary. This is a value is randomized and unique for each thread. This value is checked when a function returns and, if it has been modified, the program aborts. Unless an attacker can exploit an infoleak to obtain the actual value of the canary, any overflow will blindly overwrite it and trip the SSP protection.

• NX (No Execute), also called DEP (Data Execution Prevention) on Windows, is a hardware security feature that must be enabled at compile-time to be supported for any given application. When NX is active, memory that is marked as readable by not executable cannot be executed. Without NX, readability implies executability. Most operating systems use NX globally, preventing naive buffer overflows. Bypassing NX usually requires more advanced exploitation techniques, such as ROP (a technique to execute existing code out-of-order).

There are other common hardening techniques, such as RELRO, BIND_NOW, FORTIFY_SOURCE, as well as less common techniques like UBSAN, CFI, and SafeStack. The popular checksec utility is able to check for the presence of several hardening features in any executable or process. It runs on almost any Linux system. None of these are perfect, but they can significantly raise the bar for an attacker. Different operating systems, and even different distributions within the same operating system, may apply differing levels of hardening.

Kernel hardening

Kernel hardening tends to be most important when an attacker is running a malicious process on your system. This can be the case if either you are legitimately permitting them to run an untrusted process (in a virtual machine or container, for example), as well as if they have exploited an application with low privileges and wish to elevate themselves to higher privileges, such as root. It is generally harder for an attacker to get into a position where they can attack the kernel, but if they can, the impact is significant (complete loss of integrity).

The kernel is the lowest-level, most privileged task in any operating system. Userspace processes communicate with the kernel through syscalls in order to request various resources (such as memory) or actions (such as I/O). The kernel is in charge of deciding whether or not the calling userspace process has sufficient permissions. Unfortunately, the interface to the kernel is very complex, and any bug in this interface can completely invalidate the integrity of the running operating system. The kernel thus needs to protect itself. This is called kernel hardening.

On Linux systems, kernel security can be increased greatly by using the grsecurity patchsets. Unfortunately, as of writing this, the patchsets are commercial. If you do purchase them though, you can get improvements that are very possibly decades ahead of public security techniques, if history is to be trusted (ASLR and NX for example came from research done for PaX, one of the core components of grsecurity which focuses on resisting memory corruption attacks).

Differences between categories of operating systems

Different operating systems may have different priorities for kernel security. For example, Debian is designed to work on virtually all hardware and with all software, even outdated software. As such, it enables potentially insecure legacy features that have a history of exploitation. OpenBSD, on the other hand, quickly removes any feature that they think is too risky to securely maintain.

A summary of a few common categories of secure operating systems:

• Linux has a kernel which is often configured differently by different distros. A stripped-down, minimal Linux kernel can be incredibly secure, whereas a feature-rich and bloated kernel may be running many exploitable drivers. The kernel is open source, so different distributions even have their own patchsets to the kernel. The Linux kernel doesn't have many self-protection features, but they are being slowly added upstream, and these features are typically based on (read: copy-pasted from) the latest publicly available grsecurity sources.

  1. A custom kernel has the potential to be the most secure. It can be compiled with hardening, and extra features can be stripped out. Unused drivers can be removed, reducing attack surface area. Security features that are mislabeled as debugging features (such as credential sanity checking) can be enabled. A custom kernel is the hardest to maintain, as it requires knowledge of kernel configuration and security. Even a basic overview of security-related configuration options would take up an entire answer.

  2. The Fedora team takes security more seriously than most, it seems. They enable new security features when they come out, and they disable old, legacy functionality. SELinux is enabled by default, leveraging the kernel's ability to restrict even the root user.

  3. Ubuntu used to be fairly insecure but has been improving security as of late. Their kernel supports some legacy features, but also disables others which may conflict with security policies. Ubuntu is Debian-based, but uses its own kernel. Userspace has a decent amount of hardening, with PIE and other techniques applied to a number of utilities.

  4. Debian, unfortunately, prioritizes compatibility over security. Their kernel supports a large number of potentially risky legacy functionality (uselib, vsyscalls, etc) in the name of compatibility with legacy software. It has all sorts of drivers for many types of hardware, some of which can be exploited. Compile-time hardening for the kernel is virtually non-existent, and is limited to only sensitive programs for userland. However, the Debian security team is fairly quick with security backports.

• BSD is a class of operating systems derived from the original Berkeley Software Distribution. Unlike Linux distros, BSD distros do not use the same kernel. Their only similarity is their ancestor, BSD 4.4. Security in the BSDs varies greatly:

  1. OpenBSD is a BSD operating system that prides itself on security and stability. Many features that may be insecure are stripped out, and many new security techniques are added regularly. It is designed to be secure by default, with almost all of the userland sandboxing itself and compiled with extensive protections. The kernel itself is not perfect and fuzzing it can reveal some scary results. The system doesn't have the best performance, in part because of their focus on security. This is especially apparent on multiprocessor systems, where the majority of the kernel is forced into a single thread by the giant lock to reduce the risk of race conditions.

  2. FreeBSD is more similar to Linux in that it often breaks away from the original Unix philosophy. It supports a large amount of hardware and has very good software support. The kernel is not quite as clean as OpenBSD's, and its userland is not ideal (it uses the ASLR-limiting jemalloc memory allocator, for example), but it is overall a fairly solid system. There is a modified version of FreeBSD called HardenedBSD which focuses on userspace hardening, though its kernel does have some improvements, especially in regards to ASLR. It is still rather experimental last I checked, but is otherwise very solid.

  3. NetBSD and DragonflyBSD are not great. The kernels, especially the networking stack of NetBSD, is riddled with bugs. DragonflyBSD was so behind on security that they only added NX support (a hardware feature that prevents executing non-executable code) to the kernel recently, something Linux got in 2004! These are niche systems and it's unlikely you will need to use them.

• Windows, especially 10, is actually surprisingly secure. The kernel is possibly the most heavily statically-analyzed monolithic kernel in existence, and it supports a large number of security features and self-protection techniques. Older versions of Windows were so far behind on security (you could even map an address to NULL on Vista, if I recall) that it has adopted a very bad reputation. Now days, the kernel is actually quite hardened. The complexity of the kernel does lead to other bugs, but exploitation-wise, it is likely more secure than an equivalently feature-rich Linux kernel, such as Debian's. The biggest issue with Windows is the fact that that you cannot compile out features you do not need or audit the source code. Additionally, the Snowden leaks have shown that Microsoft gives the intelligence community early access to 0days before they are patched.

Security features (access controls, etc)

A secure kernel means nothing if it cannot be used to enforce security policies for userspace. What good is a parent who can't be manipulated if they don't know how to keep their kids in line? Most kernels provide various interfaces that privileged users can set up in order to restrict lesser users. Some of these features are enabled by default and are a core part of the system's security (such as standard Unix access controls, aka DAC), and some are optional and can be configured to provide fine-grained protections and sandboxing.

Security features like this are useful when there is the risk of an attacker exploiting a running application, such as a web browser or media viewer. If the individual binary hardening is not sufficient to prevent exploitation, kernel-provided security restrictions can limit the damage that can be done. For example a browser which is limited to accessing only its own configuration directory, a public directory, and the downloads directory will not be able to read your password files.

A small list of common kernel-provided security features:

• DAC (Discretionary Access Controls) are standard Unix permissions, supported on all Unix and Unix-like systems. Any individual file can be given attributes to restrict reading, writing, and executing. There are separate attributes that apply to the user (let's say fred) which owns the file (in which case they can override it at will, hence "discretionary"), the group the file is owned by (let's say localusers), and all other miscellaneous users on the system. For example, a file can be made readable and writable for its user, only readable by the group, and completely unreadable and unwritable for any other process on the system. This example permission would be fred:localusers 640, with 640 being the permissions in octal.

• MAC (Mandatory Access Controls) are present in many security-focused operating systems. Unlike DAC, even the owner of a policy cannot override it. A MAC system typically restricts the resources (such as a file or networking interface) a given subject (such as a process, program, or user) is permitted to access. Common examples are SELinux, a difficult-to-configure by highly extensible framework provided by default on Fedora and CentOS, and AppArmor, an easier to use system that focuses on limiting file access, provided by default on Debian and Ubuntu, as well as OpenSUSE.

• Sandboxes and jails can be used to restrict an untrusted process, reducing the capabilities it has and preventing it from damaging the rest of the system. The implementations, designs, and purposes vary greatly. It is a complex enough subject on its own that I can only link to a different answer which I've written, covering sandboxes and similar technologies.

Default services

The default installed applications and the services that run by default are very important for security. A stripped-down system with very few unnecessary features will be more secure than a bloated system with applications for every possible format. More running services means more attack surface area. Some operating systems, like Windows, want to run as many services as possible, whereas OpenBSD has practically nothing running by default. Because they have such a bare default install with so few running services, they can proudly boast:

Only two remote holes in the default install, in a heck of a long time!

There are countless online guides for disabling unneeded services on all sorts of operating systems. Since the base applications determine how much attack surface area the operating system has by default, a system with a huge base can present itself as low-hanging fruit to potential attackers. Ubuntu for example fell the very first day in the Pwn2Own contest due to its feature-rich userland. A ChromeOS system with its more limited system would have held up!

Recommendations

You didn't ask for recommendations, but just in case:

• ChromeOS is a Linux-based system developed by Google. It is focused on web browsing and so you should expect to spend most of your time in the browser. It is possibly the most secure Linux system out-of-the-box, as well as one of the easiest to use. It uses a built-in TPM to provide measured boot, preventing the system from coming online if it has been tampered with (either in an evil-maid attack, or by malware). Privacy-wise, it's obviously not great.

• QubesOS is a Xen-based system which uses Fedora internally. It focuses on isolation-by-default and tries to be easy to use. It is primarily developed by Joanna Rutkowska. Xen uses IOMMU isolation which prevents malicious or compromised hardware peripherals (e.g. the network card) from compromising the system. It can make use of an installed TPM to provide measured boot using what it terms AEM. Qubes makes some strange security decisions, such as using Xen (which has a poor track record for security), and giving all AppVMs root access, relying entirely on Xen for security. Its heavy use of virtualization can make it sluggish, and it does not work on systems without hardware-accelerated virtualization.

• Whonix or Tails can be used if you want to focus on anonymity. They both force all their connections through the Tor anonymity network, but differ slightly in their goals. Whonix is designed to use physical isolation, running the Tor process on another operating system (the "gateway") so that, even if the primary workstation is compromised, the Tor process is not harmed. By default, Whonix recommends using virtualization to achieve this, but this recommendation should not be taken. Instead, the gateway and workstation should be installed on physically separate machines. Tails is similar, but uses a software firewall for isolation instead. It additionally focuses on anti-forensics, keeping everything in memory and losing it when the system reboots. To accomplish this, Tails is a live DVD.

• Fedora is a Linux distribution that focuses on security and ease-of-use. Current Fedora releases use Wayland by default to improve graphical isolation. Fedora also uses fairly well-hardened executables and a kernel that is kept up to date and secure. It also uses SELinux out of the box with pre-configured policies. For a general-purpose Linux distribution, Fedora is among the most secure while still being user-friendly.

• OpenBSD is a Unix-derived operating system that focuses on simplicity, security, and stability. It has an excellent documentation system, providing probably the most comprehensive and useful references of any operating system. The man pages are in fact considered the official documentation for the system. OpenBSD is a very good choice if you are fine with the base system and other applications available through its binary package manager. Overall, OpenBSD feels very... comfy. It's what an operating system should be, even if it has its drawbacks (lower software and hardware compatibility, doesn't scale well on SMP systems).

• Windows is the most popular desktop operating system in the world. Recent versions improve security, as mentioned earlier. Windows may be useful if you do not care about privacy and want a well-supported system that has decent resistance to exploitation.

Whatever the OS, the most dangerous element for computer security lies between the keyboard and the chair back. In the use cases you describe, what immediately lights a red bulb on my mind is: recreational use 95% of the time... - with all the involved risks, mainly for browsing the interwebz: ... where there are all these naughty ads that I can't block - 5% are online banking or remote logins to my work machine possibly via VPN - which should require stronger security.

I know no way to keep something plainly secure - secure enough to trust it for my banking operations - and at the same time tolerant enough to download and test any possible sexy media or software from the wide internet.

The rule is that each time you let some file crafted by an evil guy enter into your computer, you possibly open a security trap - here again no OS will protect you against that. The least poor way is to define security levels, and ensure that nothing from the low security levels can access data stored in high security ones. It is indeed possible with any OS, but AFAIK no OS provides it transparently and out of the box.

That being said there are some hints:

• use different users for serious activity and recreational one. But be very careful for any request for priviledge elevation! At installation time (of a game, or any software), you often see something like (in Windows world) A program asks for admin priviledges, do you agree or (in Unix-Linux world) Installation of this thing requires super-user priviledge. Please use su or sudo and restart installation. In both use case, you are left with 2 options:

  • allow the admin priviledge - which allows the install code to do anything on the machine
  • disallow it - and you cannot install that damned so sexy file

• use a dedicated VM for recreational operations. It may looks nice, but VMs do not offer the full power of a real machine, and your brand new machine will behave as a 10 years old one. Not very nice for games and even for multi media browsing

• use a dedicated VM for the serious operations. Most malware do not try to look inside VMs (for now and for what I know, unsure how it will evolve because VM become more and more commons). On the Confidentiality, Integrity and Availability triad, you might trust C and I but A is still at high risk...

So IMHO the best way it to try to understand the possible consequences of what you do on your machine, and whether it is acceptable for the security you need on it. And no OS, nor any other piece of software can do that for you.

That being said, a good antivirus, a firewall and priviledge isolation are a must have, and VMs can help. But you cannot simply rely on them...

And all of the above applies to any OS.