If get this error trying to check the X-RAY button on the configuration tab of a lambda function:
- The Configuration tab failed to save. Reason: The provided execution role does not have permissions to call PutTraceSegments on XRAY
This means some permissions have to be added to the Lambda role as the error message states. The message above says the permissions will be automatically added but seems they are not.
An IAM role can perform the following x-ray actions, if allowed:
Out of the full list it the role only requires the "Put" commands so following best practices only add those two actions to my Lambda function role.
-
Effect: "Allow"
Action:
- "xray:PutTelemetryRecords"
Resource: "*"
-
Effect: "Allow"
Action:
- "xray:PutTraceSegments"
Resource: "*"
Here's the lambda role I'm using with the actions added:
While we're add it let's automatically add tracing to the lambda in CloudFormation. The Lambda Function CloudFormation properties includes something called TracingConfig:
Clicking on the tracing config shows tow modes: active and pass through with an explanation. We'll go with the default (Pass-Through):
Although this is configurable doesn't appear to actually turn tracing on. Hopefully that's coming soon. For now log into the console and click on the Lambda function configuration tab, advanced settings, and then check the box to enable X-Ray tracing. Save the function and run it to see X-ray in action.
So bummer, the title is a bit misleading but at least we were able to set up our IAM role with the necessary X-Ray permissions.
Once X-ray is turned on, click on the monitoring tab of the Lambda function and you'll see some additional information. The first display has the list of executions:
Click on one of the executions to see more details about the trace:
Finally click on one of the lines in the trace to get even more details such as code stack trace that caused an error - in this case connection to an S3 endpoint is failing (and I have a support ticket out to AWS about this because it is a random occurrence in my case):
Not that this does not necessarily increase your logging but it will provide some additional information and what I hope is that it will track the error to the source in a micro-services environment with APIs calling other APIs. When you need more details in the logs, turn up the logging as explained this example for AWS Boto Python Library: http://websitenotebook.blogspot.com/2017/07/detailed-aws-boto-library-logging.html
I am not sure what visibility Amazon has into these logs but as always be careful what you log where and who has access to see it. Especially since I see no way to delete these traces...still looking. The data is auto-deleted in 30 days and may be able to delete it using the API. http://docs.aws.amazon.com/xray/latest/devguide/xray-guide.pdf
In the case that you want Amazon to see the logs to help troubleshoot X-Ray may help. I'll let you know!
