Sharepoint - Permission Level which doesn't allow renaming file in document library

There is no single permission to control Rename option.

But, if the user has Contribution permissions without Delete Item permission, he will get Access Denied when edit the name of the file. Contributor users are not able to rename items in SharePoint if the Delete permission level is removed from the Contributors group.

It is by design and is mentioned in this official article: https://support.microsoft.com/en-us/help/2505742/contributor-users-are-not-able-to-rename-items-in-sharepoint-if-the-de.

Per my testing, it also affects in SharePoint 2016.

Update:

When user who has Contribution without Delete permission is trying to save the change of file name, he gets Access Denied:

The rename option comes with the permission to delete items. If you remove the delete item permission, the rename option will also be removed from the [...] menu.

1. Create a custom permission that is a copy of Contribute permission level
2. Remove the permission to delete items
3. Save you custom permission level as "Contribute without delete"
4. Assign the member group of the site with your new "Contribute without delete" permission, and remove edit and/or contribute from the member group.
5. Test to see that it actually works