Sunday, July 09, 2017

Detailed AWS Boto Library Logging

In my last post I explained how to turn on AWS X-RAY to log details about Lambda functions. That feature is probably most useful when you have a number of Lambda functions chained together. It doesn't change your logs it simply helps you trace which function called another to drill down to the original error (I think, still testing).

For detailed logging you still need to manage this within your code. For the AWS Boto library you can add this line to your logs (be careful where you send these logs because it can output a lot of details, some of which you may want to remain private):

boto3.set_stream_logger(name='botocore')

In my last blog post the error message simply stated the Lambda function could not reach the S3 endpoint. This seems to happen randomly. With the detailed boto library logging, it tells us there is a DNS issue reaching the endpoint.

Starting new HTTPS connection (5): firebox-private-cli-bucket-876833387914-us-west-2.s3.amazonaws.com
2017-07-10 03:30:55,224 botocore.endpoint [DEBUG] ConnectionError received when sending HTTP request.
...
ConnectionError: ('Connection aborted.', gaierror(-3, 'Temporary failure in name resolution'))

Since the error occurs in an AWS Lambda function using AWS DNS servers I submitted a support case to get some help with this issue. Having the extra logs and knowing more precisely what failed should help get the problem resolved more quickly.