Telling Linux kernel *not* to use certain CPUs

You could isolate some cpu cores from kernel scheduling using isolcpus parameter. Add this parameter to your grub.conf and reboot to take effect.

You want some processor affinity (or CPU affinity).

The relevant syscall is sched_setaffinity(2), but you should use it thru pthread_set_affinity_np(3) if you want to code your benchmarks for that.

The related command is taskset(1) and you might use it on the commands you want to benchmark (or on your shell).

^{If possible, take care that the machine is not loaded a lot by other tasks unrelated to the benchmark.}

Alternatively, use some hypervisor like Xen, and boot your Linux as a guest OS for that hypervisor. Then configure xen to use only some restricted set of CPU cores (I don't know the exact details, you need to find out). On Debian (and related) distributions, you might install packages like xen-linux-system-amd64, xen-hypervisor-amd64, xen-utils etc (I'm using on Debian/Sid xen-hypervisor-4.5-amd64, xen-linux-system-4.0.0-1-amd64, xen-utils-4.5 etc...).

There might be perhaps some way of configuring (perhaps at kernel-compilation time, or by passing specific arguments to the kernel thru Grub loader) your Linux kernel to restrict the number of usable cores.

Of course, it is better, when benchmarking some program -e.g. on a desktop Linux PC-, to take care to avoid having many outside running processes (use ps auxw, pstree -p, top to find these). At least, close and quit most interactive applications (browser like firefox, email, editor, IDEs like Eclipse) and keep only a small number of terminals to benchmark. You might even benchmark in batch mode (using batch or at to run the benchmarks) while you are not even logged in (so don't have any GUI sessions à la Gnome or KDE or Xfce, etc... running).