Why does RHEL use swap even when vm.swappiness = 1?

As you’ve been told before (see Why does swappiness not work?), changing swappiness only affects future decisions made by the kernel when it needs to free memory. Reducing it won’t cause the kernel to reload everything that’s been swapped out.

Your vmstat output shows that swap isn’t being actively used, i.e. your current workloads really don’t need the pages which have been swapped out.

There’s no point in trying to micro-manage the kernel’s use of swap in the way you tend to do. Depending on your workload, decide whether you need to favour the page cache or not, adjust swappiness accordingly, then leave the system to run.

If you really want to clear swap, disable it and re-enable it:

swapoff -a && swapon -a

free -m is not a reliable source of information about swap use. Instead, please use vmstat before and after the echo commands which temporarily change swappiness.

1) swapoff -a && swapon -a && vmstat
2) do work which requires swapping
3) vmstat

Now you know how much swapping is going on before changing the swappiness. If there's no swapping going on, find other jobs which do swap.

4) use and echo command to change swappiness temporarily
5) swapoff -a && swapon -a && vmstat
6) do work which requires swapping
7) vmstat

8) compare the si and so values.

The values to watch are:

si: Amount of memory swapped in from disk (/s).
so: Amount of memory swapped to disk (/s).

You may also find helpful information in the RHEL 7 Performance Tuning Guide.

Many thanks to Stephen Kitt for reminding me about swapon and swapoff.