Gabriel Gabriel - 24 days ago 8
Python Question

PermissionError: [Errno 13] Permission denied (after multiple successful writting attemps in the file)

I wrote a code which query a mongo database and write results in a file.
My code create the file and start to write in it succesfully. But after multiple iterations (not sure if the number of iteration is fix or not) I got a PermissionError.

I've search about it but I only found answers about people who got the error at first attempt because they don't have permission. I will precise that I am not doing anything on my computer during the execution so I really don't understand how it can happen.

Here is parts of the code:

def query(self, query_date_part, query_actKey_part, filepath):
empty = True
print("0.0 %")
for i in range(len(query_date_part)):
query = {"dt": query_date_part[i], "actKey": query_actKey_part}
cursor = self.collection.find(query)
while cursor.alive:
try:
if empty:
with open(filepath, 'w') as fp:
json.dump(cursor.next(), fp, default=json_util.default)
empty = False
else:
append_to_json(filepath, cursor.next())
except StopIteration:
print("Stop Iteration")
print(str(round(float(i+1) / len(query_date_part) * 100, ndigits=2)) + " %")
return 0


def append_to_json(filepath, data):
"""
Append data in JSON format to the end of a JSON file.
NOTE: Assumes file contains a JSON object (like a Python dict) ending in '}'.
:param filepath: path to file
:param data: dict to append
"""
# construct JSON fragment as new file ending
new_ending = ", " + json.dumps(data, default=json_util.default)[1:-1] + "}\n"
# edit the file in situ - first open it in read/write mode
with open(filepath, 'r+') as f:
f.seek(0, 2) # move to end of file
index = f.tell() # find index of last byte
# walking back from the end of file, find the index
# of the original JSON's closing '}'
while not f.read().startswith('}'):
index -= 1
if index == 0:
raise ValueError("can't find JSON object in {!r}".format(filepath))
f.seek(index)
# starting at the original ending } position, write out
# the new ending
f.seek(index)
f.write(new_ending)`


Part of the output:

6.75 %
Stop Iteration
6.76 %
Traceback (most recent call last):
File "C:/Users/username/PycharmProjects/mongodbtk/mquerytk.py", line 237, in <module>
mdbc.query(split_date(2017,5,6,1,0,2017,5,16,10,0,step=2), {"$in": ["aFeature"]}, 'test.json')
File "C:/Users/username/PycharmProjects/mongodbtk/mquerytk.py", line 141, in query
append_to_json(filepath, cursor.next())
File "C:/Users/username/PycharmProjects/mongodbtk/mquerytk.py", line 212, in append_to_json
with open(filepath, 'r+') as f:
PermissionError: [Errno 13] Permission denied: 'test.json'
Process finished with exit code 1


Note: The size of the file increase during the execution. When it crash it is about 300 Mo, I still have a lot of space on my hard drive but maybe the size of the file can be an issue ?

Config: I use Windows 7, Python 3.6 and my IDE is PyCharm Community Edition 2016.3.2

Answer Source

I had the same issue, and after testing it out, it seems like there might be some "bug" when trying to write to the same file too "often", multiple times in each sec. I'll provide a very small code snippet what you can test with:

import csv

text = "dfkjghdfkljghflkjghjkdfdfgsktjgrhsleiuthsl uirghuircbl iawehcg uygbc sgygerh"
FIELD_NAMES = ['asd', 'qwe']


with open('test.txt', 'w', newline='') as csvfile:
    writer = csv.DictWriter(csvfile, fieldnames=FIELD_NAMES)
    writer.writeheader()

max = 10000
i = 0

while i <= max:
    print(str(i))
    with open('test.txt', 'a', newline='') as csvfile:
        writer = csv.DictWriter(csvfile, fieldnames=FIELD_NAMES)
        rowData = {'asd': text, 'qwe': text}
        writer.writerow(rowData)
    i += 1

The code I think is pretty self explanatory. I got the error very randomly, sometimes it happens after the ~75th iteration, sometimes it can get to even ~750, but it looks like the code can't reach the limit. So I recommend you to try to write more data a few times rather than few data very often. I hope it helps.