myoan myoan - 3 months ago 9
Python Question

Why does setup.py sweeps the content of the namespace before installing?

I'm using namespaces with setuptools to distribute a same module in two different repositories. The goal is to get

mymodule.one
and
mymodule.two
installed, knowing that the content of
one
and
two
comes from different repos. But it looks like two
setup.py
sweep each other content.

├── repo1
│ ├── mymodule
│ │ ├── __init__.py
│ │ └── one
│ │ └── __init__.py
│ └── setup.py
└── repo2
├── mymodule
│ ├── __init__.py
│ └── two
│ └── __init__.py
└── setup.py


The namespace has the
__init__.py
below:

test$ cat repo1/mymodule/__init__.py
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

test$ cat repo2/mymodule/__init__.py
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)


The
setup.py
declares the same name:

test$ cat repo1/setup.py
#!/usr/bin/env python
from setuptools import setup, find_packages
setup(name='mymodule', packages=find_packages())

test$ cat repo2/setup.py
#!/usr/bin/env python
from setuptools import setup, find_packages
setup(name='mymodule', packages=find_packages())


Installing the module from the first package allows to import it successfully:

test/repo1$ sudo python3 setup.py install
running install
Checking .pth file support in /usr/local/lib/python3.4/dist-packages/
/usr/bin/python3 -E -c pass
TEST PASSED: /usr/local/lib/python3.4/dist-packages/ appears to support .pth files
running bdist_egg
running egg_info
writing dependency_links to mymodule.egg-info/dependency_links.txt
writing mymodule.egg-info/PKG-INFO
writing top-level names to mymodule.egg-info/top_level.txt
reading manifest file 'mymodule.egg-info/SOURCES.txt'
writing manifest file 'mymodule.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/mymodule
copying build/lib/mymodule/__init__.py -> build/bdist.linux-x86_64/egg/mymodule
creating build/bdist.linux-x86_64/egg/mymodule/one
copying build/lib/mymodule/one/__init__.py -> build/bdist.linux-x86_64/egg/mymodule/one
byte-compiling build/bdist.linux-x86_64/egg/mymodule/__init__.py to __init__.cpython-34.pyc
byte-compiling build/bdist.linux-x86_64/egg/mymodule/one/__init__.py to __init__.cpython-34.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying mymodule.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying mymodule.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying mymodule.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying mymodule.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
mymodule.__pycache__.__init__.cpython-34: module references __path__
creating 'dist/mymodule-0.0.0-py3.4.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing mymodule-0.0.0-py3.4.egg
creating /usr/local/lib/python3.4/dist-packages/mymodule-0.0.0-py3.4.egg
Extracting mymodule-0.0.0-py3.4.egg to /usr/local/lib/python3.4/dist-packages
Adding mymodule 0.0.0 to easy-install.pth file

Installed /usr/local/lib/python3.4/dist-packages/mymodule-0.0.0-py3.4.egg
Processing dependencies for mymodule==0.0.0
Finished processing dependencies for mymodule==0.0.0


Here's the import:

test/$ ipython3
Python 3.4.3 (default, Oct 14 2015, 20:28:29) [...]

In [1]: from mymodule import [TAB]
extend_path one


Now we install the other part of the namespace from the second repository:

test/repo2$ sudo python3 setup.py install
running install
Checking .pth file support in /usr/local/lib/python3.4/dist-packages/
/usr/bin/python3 -E -c pass
TEST PASSED: /usr/local/lib/python3.4/dist-packages/ appears to support .pth files
running bdist_egg
running egg_info
creating mymodule.egg-info
writing mymodule.egg-info/PKG-INFO
writing top-level names to mymodule.egg-info/top_level.txt
writing dependency_links to mymodule.egg-info/dependency_links.txt
writing manifest file 'mymodule.egg-info/SOURCES.txt'
reading manifest file 'mymodule.egg-info/SOURCES.txt'
writing manifest file 'mymodule.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/mymodule
copying mymodule/__init__.py -> build/lib/mymodule
creating build/lib/mymodule/two
copying mymodule/two/__init__.py -> build/lib/mymodule/two
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/mymodule
copying build/lib/mymodule/__init__.py -> build/bdist.linux-x86_64/egg/mymodule
creating build/bdist.linux-x86_64/egg/mymodule/two
copying build/lib/mymodule/two/__init__.py -> build/bdist.linux-x86_64/egg/mymodule/two
byte-compiling build/bdist.linux-x86_64/egg/mymodule/__init__.py to __init__.cpython-34.pyc
byte-compiling build/bdist.linux-x86_64/egg/mymodule/two/__init__.py to __init__.cpython-34.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying mymodule.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying mymodule.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying mymodule.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying mymodule.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
mymodule.__pycache__.__init__.cpython-34: module references __path__
creating dist
creating 'dist/mymodule-0.0.0-py3.4.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing mymodule-0.0.0-py3.4.egg
removing '/usr/local/lib/python3.4/dist-packages/mymodule-0.0.0-py3.4.egg' (and everything under it)
creating /usr/local/lib/python3.4/dist-packages/mymodule-0.0.0-py3.4.egg
Extracting mymodule-0.0.0-py3.4.egg to /usr/local/lib/python3.4/dist-packages
mymodule 0.0.0 is already the active version in easy-install.pth

Installed /usr/local/lib/python3.4/dist-packages/mymodule-0.0.0-py3.4.egg
Processing dependencies for mymodule==0.0.0
Finished processing dependencies for mymodule==0.0.0


But trying to import
one
again fails since
two
has trashed it:

test/$ ipython3
Python 3.4.3 (default, Oct 14 2015, 20:28:29) [...]

In [1]: from mymodule import [TAB]
extend_path two

In [1]: from mymodule import one
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-1-ddf1c613e57c> in <module>()
----> 1 from mymodule import one

ImportError: cannot import name 'one'

Answer

There are two requirements for using name spaces correctly.

  1. __init__.py of modules declare a name space
  2. setup.py defines unique name for each module

The contents of both __init__.py files should be:

__import__('pkg_resources').declare_namespace(__name__)

Then setup.py for first module:

setup(name='mymodule_one', packages=find_packages('.'), 
      namespace_packages=['mymodule'])

and second module

setup(name='mymodule_two', packages=find_packages('.'),
      namespace_packages=['mymodule'])

As a result, should be able to install and import both mymodule.one and mymodule.two

The common name space mymodule allows both modules to be imported using the same name.

The name given to setup.py for each module needs to be unique as that is used for the installation path of the module and will overwrite anything that shares it, as you have seen.