kubernetes mysql chown operation not permitted

Ok it seems I can answer my own question, the problem was lying in the NFS share that was being used as the persistent volume. I had it set to 'squash_all' in the export but it needs to have a 'no_root_squash' to allow root in case of docker container to chown on the nfs bound volume.

I solved this problem other way. I had an argument with system administrator regarding allowing root access to exported NFS directory on NFS client machine(s). He has valid security reasons for not setting it such reason one and reason two -read no_root_squash section.

At the end I didn't have to request no_root_squash. This is what I did to make mysql pod running without compromising security.

Step 1

Exec into pod's container runing mysql image. kubectl exec -it -n <namespace> <mysql_pod> -- bash

Step 2

Obtain uid (999) and gid (999) of mysql user. cat /etc/passwd | tail -n or id mysql. mysql username can be found in 2nd instruction specified in Dockerfile

Step 3

Change permission to the directory that holds content of /var/lib/mysql of docker container. This is more likely the directory specified in your PersistentVolume. This command is executed on host machine, not in the Pod!!!

# PerisistentVolume
...
nfs:
    path: /path/to/app/mysql/directory
    server: nfs-server

Run chown 999:999 -r /path/to/app/mysql/directory

Step 4

Finally after everything is set, deploy your MySQL Pod (deployment, replica set or whatever you are using).