I cant get samba to set proper permissions on created directories

I think you need to use the following parameters:

# I changes the permissions to rw-rw-r--
# You should be able to change them to 775 if you need the files to
# be executable
create mask = 664
force create mode = 664
security mask = 664
force security mode = 664

# I set the SGID flag here as I thought this is what you wanted
# You could change to 0775
directory mask = 2775
force directory mode = 2775
directory security mask = 2775
force directory security mode = 2775

I was looking for a nice explanation of how these settings work, but could not find anything better then man smb.conf

You will have to scroll down a bit for those options.

Basically, in a nutshell, windows permissions are not the same as unix (linux) and it is a bit odd how samba maps permissions.

I had the same problem, but everything like mask directives did not work for me (Samba 4.3.11):

 create mask = 0664
 force create mode = 0664
 directory mask = 02775
 force directory mode = 02775

The only option that worked was under the [global] or share section:

 inherit permissions = yes

Just change all folder and file permissions to your need, so future folders and files will inherit the same permissions.

After a lot of trial and error, this is the correct code to share samba dir using SGID and unix groups. If user connects anonymously he gets r/o, if he logs in and is a member of assigned group he gets r/w.

I have group named 'admin' set as primary group to users with write privileges, everyone else gets read only rights.

I force user to nobody, so different people working on same files don't interfere with each other.

I set chmod 2755 on shared directory, so it inherits created directories with the same group 'admin'

$ chmod -R 2755 /home/shares/test

Checking if all is good:

$ stat /home/shares/test
Access: (2755/drwxr-sr-x)  Uid: (65534/  nobody)   Gid: ( 1001/   admin)

Relevant part of /etc/samba/smb.conf:

[test]
        comment = test
        path = /home/shares/test
        force user = nobody
        read only = No
        create mask = 0664
        force create mode = 0664
        directory mask = 02775
        force directory mode = 02775

This post put me on right track, but testparm revealed 4 incorrect directives, so I'm sharing fixed config here. In samba, the less directives you specify the better it works.