What is the best way to open a file for exclusive access in Python?

The solution should work inside the same process (like in the example above) as well as when another process has opened the file.

If by 'another process' you mean 'whatever process' (i.e. not your program), in Linux there's no way to accomplish this relying only on system calls (fcntl & friends). What you want is mandatory locking, and the Linux way to obtain it is a bit more involved:

Remount the partition that contains your file with the mand option:

# mount -o remount,mand /dev/hdXY

Set the sgid flag for your file:

# chmod g-x,g+s yourfile

In your Python code, obtain an exclusive lock on that file:

fcntl.flock(fd, fcntl.LOCK_EX)

Now even cat will not be able to read the file until you release the lock.

EDIT: I solved it myself! By using directory existence & age as a locking mechanism! Locking by file is safe only on Windows (because Linux silently overwrites), but locking by directory works perfectly both on Linux and Windows. See my GIT where I created an easy to use class 'lockbydir.DLock' for that:

https://github.com/drandreaskrueger/lockbydir

At the bottom of the readme, you find 3 GITplayers where you can see the code examples execute live in your browser! Quite cool, isn't it? :-)

Thanks for your attention

---

This was my original question:

I would like to answer to parity3 (https://meta.stackoverflow.com/users/1454536/parity3) but I can neither comment directly ('You must have 50 reputation to comment'), nor do I see any way to contact him/her directly. What do you suggest to me, to get through to him?

My question:

I have implemented something similiar to what parity3 suggested here as an answer: https://stackoverflow.com/a/21444311/3693375 ("Assuming your Python interpreter, and the ...")

And it works brilliantly - on Windows. (I am using it to implement a locking mechanism that works across independently started processes. https://github.com/drandreaskrueger/lockbyfile )

But other than parity3 says, it does NOT work the same on Linux:

os.rename(src, dst)

Rename the file or directory src to dst. ... On Unix, if dst exists and is a file, it will be replaced silently if the user has permission. The operation may fail on some Unix flavors if src and dst are on different filesystems. If successful, the renaming will be an atomic operation (this is a POSIX requirement). On Windows, if dst already exists, OSError will be raised (https://docs.python.org/2/library/os.html#os.rename)

The silent replacing is the problem. On Linux. The "if dst already exists, OSError will be raised" is great for my purposes. But only on Windows, sadly.

I guess parity3's example still works most of the time, because of his if condition

if not os.path.exists(lock_filename):
    try:
        os.rename(tmp_filename,lock_filename)

But then the whole thing is not atomic anymore.

Because the if condition might be true in two parallel processes, and then both will rename, but only one will win the renaming race. And no exception raised (in Linux).

Any suggestions? Thanks!

P.S.: I know this is not the proper way, but I am lacking an alternative. PLEASE don't punish me with lowering my reputation. I looked around a lot, to solve this myself. How to PM users in here? And meh why can't I?

I don't think there is a fully crossplatform way. On unix, the fcntl module will do this for you. However on windows (which I assume you are by the paths), you'll need to use the win32file module.

Fortunately, there is a portable implementation (portalocker) using the platform appropriate method at the python cookbook.

To use it, open the file, and then call:

portalocker.lock(file, flags)

where flags are portalocker.LOCK_EX for exclusive write access, or LOCK_SH for shared, read access.

Here's a start on the win32 half of a portable implementation, that does not need a seperate locking mechanism.

Requires the Python for Windows Extensions to get down to the win32 api, but that's pretty much mandatory for python on windows already, and can alternatively be done with ctypes. The code could be adapted to expose more functionality if it's needed (such as allowing FILE_SHARE_READ rather than no sharing at all). See also the MSDN documentation for the CreateFile and WriteFile system calls, and the article on Creating and Opening Files.

As has been mentioned, you can use the standard fcntl module to implement the unix half of this, if required.

import winerror, pywintypes, win32file

class LockError(StandardError):
    pass

class WriteLockedFile(object):
    """
    Using win32 api to achieve something similar to file(path, 'wb')
    Could be adapted to handle other modes as well.
    """
    def __init__(self, path):
        try:
            self._handle = win32file.CreateFile(
                path,
                win32file.GENERIC_WRITE,
                0,
                None,
                win32file.OPEN_ALWAYS,
                win32file.FILE_ATTRIBUTE_NORMAL,
                None)
        except pywintypes.error, e:
            if e[0] == winerror.ERROR_SHARING_VIOLATION:
                raise LockError(e[2])
            raise
    def close(self):
        self._handle.close()
    def write(self, str):
        win32file.WriteFile(self._handle, str)

Here's how your example from above behaves:

>>> path = "C:\\scr.txt"
>>> file1 = WriteLockedFile(path)
>>> file2 = WriteLockedFile(path) #doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
    ...
LockError: ...
>>> file1.write("111")
>>> file1.close()
>>> print file(path).read()
111