Rififi Rififi - 3 months ago 20
Python Question

Libraries in "virtualenv" much bigger than system libraries

I use virtualenv through pew (which I think is a fantastic tool), but I noticed something strange.

I have scipy system-side installed:

7,7 MiB [##########] /sparse
5,1 MiB [###### ] /special
5,1 MiB [###### ] /stats
5,0 MiB [###### ] /linalg
3,5 MiB [#### ] /spatial
3,0 MiB [### ] /optimize
2,5 MiB [### ] /signal
2,3 MiB [### ] /interpolate
2,3 MiB [## ] /misc
2,2 MiB [## ] /io
1,5 MiB [## ] /integrate
1,3 MiB [# ] /ndimage
1,0 MiB [# ] /fftpack
744,0 KiB [ ] /cluster
512,0 KiB [ ] /odr
464,0 KiB [ ] /constants
252,0 KiB [ ] /_lib
44,0 KiB [ ] /_build_utils
36,0 KiB [ ] /__pycache__
24,0 KiB [ ] HACKING.rst.txt
12,0 KiB [ ] THANKS.txt
8,0 KiB [ ] INSTALL.rst.txt
4,0 KiB [ ] __init__.py
4,0 KiB [ ] __config__.py
4,0 KiB [ ] LICENSE.txt
4,0 KiB [ ] setup.py
4,0 KiB [ ] BENTO_BUILD.txt
4,0 KiB [ ] version.py
4,0 KiB [ ] linalg.pxd


And this is scipy virtualenv-side installed (same scipy version):

51,0 MiB [##########] /sparse
37,6 MiB [####### ] /.libs
12,9 MiB [## ] /linalg
10,6 MiB [## ] /spatial
9,7 MiB [# ] /special
6,0 MiB [# ] /interpolate
5,9 MiB [# ] /stats
5,1 MiB [# ] /optimize
4,2 MiB [ ] /signal
3,2 MiB [ ] /io
3,0 MiB [ ] /integrate
3,0 MiB [ ] /ndimage
2,3 MiB [ ] /misc
2,1 MiB [ ] /cluster
1,7 MiB [ ] /fftpack
884,0 KiB [ ] /odr
328,0 KiB [ ] /constants
204,0 KiB [ ] /_lib
32,0 KiB [ ] /_build_utils
24,0 KiB [ ] HACKING.rst.txt
20,0 KiB [ ] /__pycache__
12,0 KiB [ ] THANKS.txt
8,0 KiB [ ] INSTALL.rst.txt
4,0 KiB [ ] __init__.py
4,0 KiB [ ] LICENSE.txt
4,0 KiB [ ] setup.py
4,0 KiB [ ] __config__.py
4,0 KiB [ ] BENTO_BUILD.txt
4,0 KiB [ ] version.py
4,0 KiB [ ] pip-delete-this-directory.txt
4,0 KiB [ ] linalg.pxd


Needless to say there is a huge size difference. It would normally not bother me much, but I'm trying to bundle an executable file with pyinstaller and the resulting executable file is unrealistically too big.

Can someone explain such a difference ? It is not specific to scipy, I also see it for numpy, and maybe for other libraries.

EDIT:

The files inside the directories have different sizes:

System-wide:

3,1 MiB [##########] _sparsetools.cpython-35m-x86_64-linux-gnu.so


Virtualenv-wide:

38,5 MiB [##########] _sparsetools.cpython-35m-x86_64-linux-gnu.so

Answer

The shared library files in the Python wheels distributed for Scipy aren't stripped, so they are bigger than what your package manager installs:

$ file _sparsetools.cpython-35m-x86_64-linux-gnu.so
_sparsetools.cpython-35m-x86_64-linux-gnu.so: ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=ed7b8e00c558f234620423315fa9b53274393e72, not stripped
$ du -h _sparsetools.cpython-35m-x86_64-linux-gnu.so
39M     _sparsetools.cpython-35m-x86_64-linux-gnu.so

If you strip it, the file size shrinks:

$ strip _sparsetools.cpython-35m-x86_64-linux-gnu.so
$ file _sparsetools.cpython-35m-x86_64-linux-gnu.so
_sparsetools.cpython-35m-x86_64-linux-gnu.so: ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=ed7b8e00c558f234620423315fa9b53274393e72, stripped
$ du -h _sparsetools.cpython-35m-x86_64-linux-gnu.so
3.7M    _sparsetools.cpython-35m-x86_64-linux-gnu.so

PyInstaller can do this for you with the --strip flag.

Comments