I have a long-running worker process running on EC2 that consumes items from an SQS queue. After a time (8-12 hours, I reckon) I begin getting expired security token errors. I would expect the aws lib to handle the refresh of credentials automatically but this seems not to be the case. Is it in anyway handled within the client?
This happens only when I use the DefaultCredentialsProviderChain to generate the access. This error does not occur when used with key and secret.
The stacktrace is as below:
com.amazonaws.AmazonServiceException: The security token included in the request is expired (Service: AmazonSQS; Status Code: 403; Error Code: ExpiredToken; Request ID: 6ff6e1a0-d668-5ac5-bcd7-ae30058f25c0)
awsCredentials = (new DefaultAWSCredentialsProviderChain).getCredentials
sqs = SimpleSQSClient(awsCredentials, Regions.US_EAST_1)
queueSQS = sqs.simple(QueueName(queueName), true)
The AWS SDKs are indeed capable of cycling temporary credentials inherited from the instance profile, but by passing an explicit AWSCredentials object in the constructor of SimpleSQSClient I believe you are denying it the opportunity to do so.
You didn't explicitly state that your application is inheriting the instance role, but there is enough evidence in your post to infer that this is the case:
The specific behavior for automatic credentials refresh is described in the documentation:
The automatic credentials refresh happens only when you use the default client constructor, which creates its own InstanceProfileCredentialsProvider as part of the default provider chain, or when you pass an InstanceProfileCredentialsProvider instance directly to the client constructor. If you use another method to obtain or pass instance profile credentials, you are responsible for checking for and refreshing expired credentials.
By passing AWSCredentials directly instead of an AWSCredentialsProvider, you become responsible for checking and refreshing expired credentials. On the plus side, your workaround is just fine if you want to keep passing credentials explicitly.
SimpleSQSClient has a constructor that will work better for your use case:
new SimpleSQSClient( credentialProvider: com.amazonaws.auth.AWSCredentialsProvider, region: com.amazonaws.regions.Regions, buffered: Boolean )
SimpleSQSClient sqs = SimpleSQSClient(new DefaultAWSCredentialsProviderChain(), Regions.US_EAST_1, false)
Example, explicitly using InstanceProfileCredentialsProvider:
SimpleSQSClient sqs = SimpleSQSClient(new InstanceProfileCredentialsProvider(), Regions.US_EAST_1, false)