Ev. Kounis Ev. Kounis - 3 months ago 28
Python Question

Copying and merging directories excluding certain extensions

Good people of SO, my problem is that i want to copy multiple directories with identical structure (subdirectories have the same names) but different contents into a third location and merge them. At the same time, i want to ignore certain file extensions and not copy them.




I found that the first task alone can be easily handled by
copy_tree()
function from the
distutils.dir_util
library. The issue here is that the
copy_tree()
cannot ignore files; it simply copies everything..


distutils.dir_util.copy_tree() - example


dirs_to_copy = [r'J:\Data\Folder_A', r'J:\Data\Folder_B']
destination_dir = r'J:\Data\DestinationFolder'
for files in dirs_to_copy:
distutils.dir_util.copy_tree(files, destination_dir)
# succeeds in merging sub-directories but copies everything.
# Due to time constrains, this is not an option.





For the second task (copying with the option of excluding files) there is the
copytree()
function from the
shutil
library this time. The problem with that now is that it cannot merge folders since the destination directory must not exist..


shutil.copytree() - example


dirs_to_copy = [r'J:\Data\Folder_A', r'J:\Data\Folder_B']
destination_dir = r'J:\Data\DestinationFolder'
for files in dirs_to_copy:
shutil.copytree(files, destination_dir, ignore=shutil.ignore_patterns("*.abc"))
# successfully ignores files with "abc" extensions but fails
# at the second iteration since "Destination" folder exists..


Is there something that provides the best of both worlds or do i have to code this myself? Thanks and sorry for the long post.

Answer

As PeterBrittain suggested, writing my own version of shutil.copytree() was the way to go. Below is the code. Note that the only difference is the wrapping of the os.makedirs() in an if block.

from shutil import copy2, copystat, Error, ignore_patterns
import os


def copytree_multi(src, dst, symlinks=False, ignore=None):
    names = os.listdir(src)
    if ignore is not None:
        ignored_names = ignore(src, names)
    else:
        ignored_names = set()

    # -------- E D I T --------
    # os.path.isdir(dst)
    if not os.path.isdir(dst):
        os.makedirs(dst)
    # -------- E D I T --------

    errors = []
    for name in names:
        if name in ignored_names:
            continue
        srcname = os.path.join(src, name)
        dstname = os.path.join(dst, name)
        try:
            if symlinks and os.path.islink(srcname):
                linkto = os.readlink(srcname)
                os.symlink(linkto, dstname)
            elif os.path.isdir(srcname):
                copytree_multi(srcname, dstname, symlinks, ignore)
            else:
                copy2(srcname, dstname)
        except (IOError, os.error) as why:
            errors.append((srcname, dstname, str(why)))
        except Error as err:
            errors.extend(err.args[0])
    try:
        copystat(src, dst)
    except WindowsError:
        pass
    except OSError as why:
        errors.extend((src, dst, str(why)))
    if errors:
        raise Error(errors)