unclemeat unclemeat - 28 days ago 31
Python Question

AWS Elasticsearch Service Role Permissions

I'm trying to submit documents to an Elasticsearch index from a Python Lambda script running as a role which has the AmazonESFullAccess policy attached. I am creating the requests with the python elasticsearch library, and signing them with the aws_requests_auth library.

cred = boto3.session.Session().get_credentials()
es_host = '<elasticsearch-server>'

auth = AWSRequestsAuth(
aws_access_key=cred.access_key,
aws_secret_access_key=cred.secret_key,
aws_host=es_host,
aws_region='us-east-1',
aws_service='es')

ES_CLIENT = Elasticsearch(
host=es_host,
port=80,
connection_class=RequestsHttpConnection,
http_auth=auth)


Then sending bulk create requests as follows:

ES_CLIENT.bulk(
index='test_index',
body=docs)


This is failing with the following:


TransportError(403, u'{"message": "The security token included in the
request is invalid." }'): AuthorizationException ...


Though the same code works when running with admin user access keys.

Why do these requests fail when running as a role with 'full ES access'?

Answer

Because Elasticsearch is a seperate entity to AWS, but a version is hosted by AWS it seems that elasticsearch doesn't treat IAM the same way you'd expect.. AWS has something in place that tries, but doesn't quite work with just an access_key and secret_key. It needs the session Token as well.

The answer to this is to use the cred.token along with the access key and secret key and then pass it into your AWSRequestsAuth object:

auth = AWSRequestsAuth(
    aws_access_key=cred.access_key,
    aws_secret_access_key=cred.secret_key,
    aws_token=cred.token
    aws_host=es_host,
    aws_region='us-east-1',
    aws_service='es')