jb. jb. - 3 months ago 17
Python Question

Google App Engine bulkloader issue when using yaml autogenerated configuration and entities with numeric ID

Please be kind as I'm new to Python and Google App Engine.

My application uses Django non-rel. I don't have access to model.

I have my

bulkloader.yaml
file autogenerated by
appcfg.py create_bulkloader_config
.

Problem is entities numeric ID's are being imported as string key names. So if I export entity with int ID of, for example, '62', it gets imported as entity with string key name of '61' which screws up Django.

Revelant bulkloader.yaml Fragment:

property_map:
- property: __key__
external_name: key
export_transform: transform.key_id_or_name_as_string


I'm trying to setup download/upload od data using bulkloader, and I want to have data as easy to understand format (like .csv) --- so using
bulkloader.py --dump (...)
is not a viable option since it gives me sqlite3 files that have entities contents pickled as a single row.

EDIT



I tried doing what @Nick suggested and I got an exception:

ErrorOnTransform: Numeric keys are not supported on input at this time.


Does this mean that I have to stick to bulkloader.py (that uses that werid sqlite format) or I messed something? ;)

Header of Transformer:

- kind: auth_user
connector: csv
connector_options:
encoding: utf-8
skip_import_header_row: True
print_export_header_row: True

property_map:
- property: __key__
external_name: key
export_transform: transform.key_id_or_name_as_string
import_transform: transform.create_foreign_key('auth_user', key_is_id=True)


Whole Stacktrace:

Traceback (most recent call last):
File "/opt/google/appengine/google/appengine/tools/adaptive_thread_pool.py", line 150, in WorkOnItems
status, instruction = item.PerformWork(self.__thread_pool)
File "/opt/google/appengine/google/appengine/tools/bulkloader.py", line 693, in PerformWork
transfer_time = self._TransferItem(thread_pool)
File "/opt/google/appengine/google/appengine/tools/bulkloader.py", line 848, in _TransferItem
self.content = self.request_manager.EncodeContent(self.rows)
File "/opt/google/appengine/google/appengine/tools/bulkloader.py", line 1269, in EncodeContent
entity = loader.create_entity(values, key_name=key, parent=parent)
File "/opt/google/appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 385, in create_entity
return self.dict_to_entity(input_dict, self.bulkload_state)
File "/opt/google/appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 131, in dict_to_entity
instance = self.__create_instance(input_dict, bulkload_state_copy)
File "/opt/google/appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 209, in __create_instance
'Numeric keys are not supported on input at this time.')

Answer

You've got the export_transform 'key_id_or_name_as_string', but you don't have a corresponding import transform. Try this:

property_map:
 - property: __key__
   external_name: key
   export_transform: transform.key_id_or_name_as_string
   import_transform: transform.create_foreign_key('Kind', key_is_id=True)

Where 'Kind' is the name of the kind identified in the config file.