Permission denied when accessing VirtualBox shared folder when member of the vboxsf group

I had added my user to the vboxsf group:

sudo usermod -aG vboxsf $(whoami)

I did do a restart, but after logging out and in again, I got access! Restarting after this and it still works. Go figure.