Torxed Torxed - 4 months ago 29
Python Question

On Windows, how to open for writing a file already opened for writing by another process?

I'm trying to open a logfile which is kept open by another process and remove the first few lines.
On Unix I'd simply do a

os.open('/tmp/file.log', os.O_NONBLOCK)
and that would get me closer to my goal.

Now i'm stuck with Windows and I need to rotate this log somehow without ending the application holding the file. Is this even possible?

At first I considered opening a file handle on the location where the application expected the log to be and just act as a pipe into a file-handle in Python but I couldn't find any way of doing that either on Windows.

I also thought of just moving the file on a regular basis and letting the application recreate the file but since it's being used by another process that doesn't do much good.

Thought of
O_SHLOCK
as well but then again, that's Unix and not Windows.
So I went for mmap the file and hope that it would make it a bit more flexible but that led me nowhere.

import mmap
import contextlib
import time

with open(r'test.log', 'r+') as f:
with contextlib.closing(mmap.mmap(f.fileno(), 0)) as m:
while 1:
line = m.readline()
if len(line) > 0:
print line
time.sleep(0.5)


This results in that the application can't access the file because Python is holding it (and vice versa).

Came to think of
signal.SIGHUP
but that doesn't exist in Windows either so back to square one.

I'm stuck and I've tried it all, can Python help me here or do I need to switch my language?

Answer

Even if the application opens the file as a shared object Python can't so they can't get along by the looks of it.

It's not so bad :). You can (have to) open a file using CreateFile as pointed out by Augusto. You can use standard ctypes module for this. In the question Using a struct as a function argument with the python ctypes module you can see how to do it. Then you have to associate a C run-time file descriptor with an existing operating-system file handle you obtained in the previous step. You can use _open_osfhandle from the MS C run-time library (CRT) to do this. You can call it once again using ctypes; you can access it as ctypes.cdll.msvcrt._open_osfhandle. Then you have to associate Python file object with an existing C run-time file descriptor you obtained in the previous step. To do this in Python 3 you simply pass file descriptor as the first argument to the built-in open function. According to docs

file is either a string or bytes object giving the pathname (absolute or relative to the current working directory) of the file to be opened or an integer file descriptor of the file to be wrapped.

In Python 2 you have to use os.fdopen; its task, according to docs, is to

Return an open file object connected to the file descriptor fd

All of the above should not be required to do such a simple thing. There's hope it will be much simpler when CPython's implementation on Windows starts using native Windows API for files instead of going through C run-time library which does not give access to many features of Windows platform. See Add new io.FileIO using the native Windows API issue for details.