Execution of possibly harmful program on Linux

• Virtual machine can give you highest security without reboot, but lowest performance.

• Another option, for even higher security than a virtual machine: boot a "live" CD/DVD/pendrive without access to the hard drive (temporarily disable the HDD in BIOS; if you can't, at least do not mount the drive / unmount it, if mounted automatically - but this is much less secure)

• A docker container is a bit less secure alternative to a full virtual machine. Probably the crucial difference (in terms of security) between these two is that systems running in docker actually use the kernel of your host system.

• There are programs such as isolate that will create a special, secured environment - this is generally called a sandbox - those are typically chroot-based, with additional supervision - find one that fits you.

• A simple chroot will be least secure (esp. in regards to executing programs), though maybe a little faster, but... You'll need to build/copy a whole separate root tree and use bind mounts for /dev etc. (see Note 1 below!). So in general, this approach cannot be recommended, especially if you can use a more secure, and often easier to set up, sandbox environment.

Note 0: To the aspect of a "special user", like the nobody account: This gives hardly any security, much less than even a simple chroot. A nobody user can still access files and programs that have read and execute permissions set for other. You can test it with su -s /bin/sh -c 'some command' nobody. And if you have any configuration/history/cache file accessible to anybody (by a mistake or minor security hole), a program running with nobody's permissions can access it, grep for confidential data (like "pass=" etc.) and in many ways send it over the net or whatever.

Note 1: As Gilles pointed in a comment below, a simple chroot environment will give very little security against exploits aiming at privilege escalation. A sole chroot makes sense security-wise, only if the environment is minimal, consisting of security-confirmed programs only (but there still remains the risk of exploiting potential kernel-level vulnerabilities), and all the untrusted programs running in the chroot are running as a user who does not run any process outside the chroot. What chroot does prevent against (with the restrictions mentioned here), is direct system penetration without privilege escalation. However, as Gilles noted in another comment, even that level of security might get circumvented, allowing a program to break out of the chroot.

Use a virtual machine. Anything less doesn't provide much security.

A few years ago I might have suggested a chrooted dedicated user or some such. But hardware has become more powerful, and virtual machine software has become easier to use. Furthermore, off-the-shelf attacks have become more sophisticated. There is no longer any reason not to go the whole way here.

I would recommend running VirtualBox. You can set up the virtual machine in a couple of minutes, then install a Linux distribution inside it. The only non-default setup I recommend is the networking setup: create both a “NAT” interface (to communicate with the world) and a “host only” interface (so you can easily copy files to and from the host, and ssh into the VM). Disable the NAT interface while you run your students' programs¹; enable it only when you're installing or upgrading software packages.

Inside the virtual machine, create one user per student.

¹ _{You can restrict the NAT interface to a whitelist of users, but that's more advanced than you need in a simple, to-the-point setup.}

here is a very thorough explanation on why using Chroot is still a very viable option, and why full operating system or full hardware virtualization is especially overkill in specific scenarios.

it is nothing more than a myth that Chroot is not a security feature. there are tools that will build the chroot file system automatically for you, and Chroot is built into many mainstream applications as a purposeful security feature.

contrary to popular belief, not every situation requires full virtualization of the operating system, or full simulation of hardware. this can actually mean having more attack surface to try and cover. in turn, meaning a less secure system. (purportedly for less-knowledgeable system administrators)

the rules are fairly simple: do not put anything inside the chroot that isn't necessary. do not run a daemon as root. do not run a daemon as any user running a daemon outside of the chroot.

remove any insecure applications, setuid binaries, dangling ownerless symlinks /hardlinks. remount unnecessary folders using nosuid, noexec and nodev. build the latest stable version of the running daemon from source. and and most of all, secure the base system!