how does fileless malware work on linux?

Fileless malware attacks the target by exploiting a vulnerability e.g. in a browser's Flash plugin, or in a network protocol.

A Linux process can be modified by using the system call ptrace(). This system call is usually used by debuggers to inspect and manage the internal state of the target process, and is useful in software development.

For instance, let's consider a process with PID 1234. This process' whole address space can be viewed in the pseudo filesystem /proc at the location /proc/1234/mem. You can open this pseudofile, then attach to this process via ptrace(); after doing so, you can use pread() and pwrite() to write to the process space.

char file[64];
pid = 1234;

sprintf(file, "/proc/%ld/mem", (long)pid);
int fd = open(file, O_RDWR);
ptrace(PTRACE_ATTACH, pid, 0, 0);

waitpid(pid, NULL, 0);
off_t addr = ...; // target process address

pread(fd, &value, sizeof(value), addr);
// or
pwrite(fd, &value, sizeof(value), addr);

ptrace(PTRACE_DETACH, pid, 0, 0);
close(fd);

(Code taken from here. Another paper about a ptrace exploit is available here.)

Concerning kernel-oriented defense against these attacks, the only way is to install kernel vendor patches and/or disabling the particular attack vector. For instance, in the case of ptrace you can load a ptrace-blocking module to the kernel which will disable that particular system call; clearly this also makes you unable to use ptrace for debugging.