How to prevent a specific program from swapping?
I think that the hiccup problem is not necessarily caused by swapping. If a program is playing something, Linux should notice this and not swap it. Programs that don't do much are the first ones that get swapped. You can check if the program is really getting swapped by looking at the RSS/RES field from ps or top. RSS is the resident set size, the non-swapped physical memory that a task is using (in kiloBytes).
I think that your problem is most probably caused by improper CPU and I/O scheduling and a bit of inefficiency of Rhythmbox which makes it sensitive to high system loads. The CPU priority can be changed with the commands nice and renice. The I/O priority can be changed with the ionice command. Only the super user can use high priorities. You should also know that Linux kernel guys are trying to improve the responsiveness of desktop systems with various low-latency patches, so you might consider using them. One of them is a ~200 lines patch written by Mike Galbraith which has impressed even Linus. The alternative to this patch is Lennart Poettering's cgroups trick which I think will be the default in Fedora 15.
Anyway, without those patches there are two options: start the program with a high priority or change it afterwards. For the first option you could use a wrapper script around Rhythmbox:
#!/bin/sh
# Run Rhytmbox with high CPU and I/O priorities
nice -n -10 ionice -c 1 -n 1 su -l -c rhythmbox alexei
You will need to run it as root. If you don't want to login as root just to start this, you can use either su or sudo.
As for changing the priority afterwards, if you're too lazy to login as root to change it, you could try using a cron job that runs every 5 minutes and sets the priority of the rhythmbox process, but I wouldn't recommend doing this:
#!/bin/sh
renice -n -10 -p `pidof rhythmbox`
ionice -c 1 -n 1 -p `pidof rhythmbox`
Short answer: You can't, and shouldn't.
A long time ago executable files honored the sticky bit +t which would tell the kernel not to swap, but today it is ignored.
If the kernel decides it has to swap, it sure has a valid reason. Linux is very aggressive on memory usage, because RAM that is idle, is a wasted resource.
If you really don't want to swap, get more RAM, or just # swapoff -a (not recomended, can turn your system unusable if you already have problems).
Shouldn't goes when you are developing some app and don't want it to swap at all. Take a look at this post on stackoverflow.
There are several ways to do that. You can attempt to try is "says" to Linux work less with swap (generally):
echo 10 > /proc/sys/vm/swappiness
From : https://www.kernel.org/doc/Documentation/sysctl/vm.txt
swappiness
This control is used to define how aggressive the kernel will swap memory pages. Higher values will increase agressiveness, lower values decrease the amount of swap.
The default value is 60.
Other option is use the cgroups kernel manager , this is per-process specific but you will have some "work" to do: Answered here: https://unix.stackexchange.com/questions/10214/per-process-swapiness-for-linux#10227