Nilesh Nilesh - 6 months ago 80
Python Question

After python to exe conversion botocore.exceptions.DataNotFoundError

I can upload data without any issue using python 2.7 interpreter on windows.
But after I compile it to exe, it shows error at line :

s3 = session.client('s3')

import HTMLParser
import ConfigParser
# Above packages necessary because boto3 depend on them
import boto3
from boto3.session import Session
# variable initialization
session = boto3.session.Session()
s3 = session.client('s3') #Error at this line
s3.upload_file(fileToUpload, bucketName, keyName)

Error is:

Traceback (most recent call last):
File "", line 39, in <module>
File "boto3\session.pyc", line 199, in client
File "botocore\session.pyc", line 754, in create_client
File "botocore\session.pyc", line 660, in get_component
File "botocore\session.pyc", line 774, in get_component
File "botocore\session.pyc", line 174, in <lambda>
File "botocore\session.pyc", line 453, in get_data
File "botocore\loaders.pyc", line 119, in _wrapper
File "botocore\loaders.pyc", line 364, in load_data
botocore.exceptions.DataNotFoundError: Unable to load data for: _endpoints

What might be the issue?


It is because boto3 could not find its data files.

In this case Python27\Lib\site-packages\botocore\data contains data files. We need to instruct boto3 to search them within data folder in the same directory where our main module is. Then we need to ask py2exe to bundle referenced packages, and data files with EXE.

Provided the data files

copy C:\Python27\Lib\site-packages\botocore\data  <project_root>\data
xcopy /s /i C:\Python27\Lib\site-packages\botocore\data\s3  <project_root>\data\s3

Provided the cacert.pem

copy C:\Python27\Lib\site-packages\botocore\vendored\requests\cacert.pem <project_root>\data

And then re-wrote my code as:

import boto3
from boto3.session import Session

CACERT = 'data/cacert.pem'
session = boto3.session.Session()
session._loader.search_paths.append('data') # boto3 will search for data files in the 'data' folder within current directory
s3 = session.client('s3', use_ssl=True, verify=CACERT) # cacert.pem will be used for SSL
s3.upload_file(fileToUpload, bucketName, keyName)

Now my python module uses resources inside the <project_root>\data instead of C:\Python27\Lib\site-packages\botocore\data

Finally, the which instructs py2exe to include all data files and referenced packages into distribution.

from distutils.core import setup
import py2exe

aws_data_files = [
                    ('data', ['data/_endpoints.json','data/_retry.json', 'data/cacert.pem']),
                    ('data/s3/2006-03-01', ['data/s3/2006-03-01/service-2.json'])
        options = {
            'py2exe': {
                'bundle_files': 1,
                'compressed': True,
                'packages': ['HTMLParser', 'ConfigParser', 'boto3.s3.inject'],
        zipfile = None,
        data_files = aws_data_files,

Answer is extracted from this issue thanks to jamesls and rayluo.