splinter splinter - 4 months ago 25
Python Question

Incorrect file reading when using os.walk in python3

I am crawling through folders using the

method. In one of the folders, there is a large number of files, around 100,000 of them. The files look like:
. But they are read as
. Indeed, when I open windows explorer to browse the folder, for the first several seconds the files look like
, but then change their appearance to
. This is a strange scenario.

Now, I can't use
because all folders and and files are being read into python variables in the looping line. Here is a snippet of the code:

for root, dirs, files in os.walk(srcFolder):
for file in files:
shutil.copy(file, storeFolder)

In the last line, I get a file not found exception, saying that the file
does not exist. Has anyone run into this mysterious issue? Anyway to bypass this? What is the cause of this? Thank you.


You don't seem to be concatenating the actual folder name with the filenames. Try changing your code to:

for root, dirs, files in os.walk(srcFolder):
    for file in files:
        shutil.copy(os.path.join(root, file), storeFolder)

os.chdir should be avoided like the plague. For one thing - if the changes suceeeds, it won't be the directory from which you are running your os.walk anymore - and then, a second chdir on another folder will fail (either stop your porgram or change you to an unexpected folder).

Just add the folder name as prefixes, and don't try using chdir.

Moreover, as for the comment from ShadowRanger above, os.walk officially breaks if you chdir inside its iteration - https://docs.python.org/3/library/os.html#os.walk - that is likely the root of the problem you had.