marioishikawa marioishikawa - 3 months ago 18
Python Question

Invalid credentials - 401 when trying to access Google Datastore from a GCE instance

I'm trying the adams.py example found on this URL
https://cloud.google.com/datastore/docs/getstarted/start_python/

As it's explained in this URL:


If you are running this code on Google Compute Engine, you do not have to create new service account credentials because the default service account for your project is used when you create an instance with the datastore and userinfo-email scopes.


While creating my instance, I checked the option:


Allow API access to all Google Cloud services in the same project.


When I run


python adams.py my-instance


I get:

ERROR:root:Error while doing datastore operation
ERROR:root:RPCError: beginTransaction Invalid Credentials
ERROR:root:HTTPError: 401 Unauthorized


In fact I get the same message even if I use a wrong name for the instance.

My problem seems similar to this thread from one year ago: Connect google datastore from existing google compute engine in python

In my own code, it works fine accessing BigQuery and then when it comes to Datastore it raises the same error, but at least is more specific addressing the line, which is the last one: resp = datastore.commit(req). Here is the section of the code use Datastore.

req = datastore.CommitRequest()
req.mode = datastore.CommitRequest.NON_TRANSACTIONAL
dsDashboard = req.mutation.insert_auto_id.add()

path_element = dsDashboard.key.path_element.add()
path_element.kind = 'Dashboard'
fmt = '%Y-%m-%d'
dashboardDate = dsDashboard.property.add()
dashboardDate.name = 'Date'
dashboardDate.value.string_value = dtDay.strftime(fmt)

dashboardValue = dsDashboard.property.add()
dashboardValue.name = 'Value'
dashboardValue.value.indexed = False
dashboardValue.value.string_value = encode(jDashboard)

resp = datastore.commit(req)


The error for my code:

Traceback (most recent call last):
File "code.py", line 262, in <module>
main()
File "code.py", line 256, in main
resp = datastore.commit(req)
File "/usr/local/lib/python2.7/dist-packages/googledatastore/__init__.py", line 90, in commit
return get_default_connection().commit(request)
File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py", line 135, in commit
datastore_v1_pb2.CommitResponse)
File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py", line 195, in _call_method
raise RPCError(method, response, content)
googledatastore.connection.RPCError: commit RPC client failure with HTTP(401) Unauthorized: Invalid Credentials


In short, it seems something simple to do. Anyone here had the same problem and has found a workaround?

Answer

The issue here is:

While creating my instance, I checked the option:

Allow API access to all Google Cloud services in the same project.

One would assume that this would make your instance get all scopes but It seems it is not working so. I've tried it and if you inspect a newly created instance with:

gcloud compute instances describe my-instance

you'll see that it only gets one scope

scopes:
  - https://www.googleapis.com/auth/cloud-platform

The solution here is to either

  • manually enable "User info" and "Cloud Datastore" ("Management, disk, networking, access & security options" > "Access & security") when creating your instance via Developers console
  • make sure you use correct scopes flag (--scopes datastore,userinfo-email) when using gcloud command as explained in Getting started with Google Cloud Datastore and Python/Protobuf article