Thread by @BenReser

I found a security vulnerability in Amazon Managed Workflows for Apache Airflow (MWAA) it's been fixed so now I can talk about it. Specifically there are two API calls that the service uses to convert AWS IAM credentials into tokens that can be used to login to Airflow. #AWS

The CreateCliToken and CreateWebLoginToken APIs were logging the tokens to Cloudtrail. The event included the hostname for the airflow server, so everything required to login to Airflow was in the event.

The issue is largely mitigated by the fact that the tokens are only valid for 60 seconds and CloudTrail delivers logs on average about every 15 minutes. So you should not receive the logs before the tokens have expired.

I reported this issue on May 11th and it was fixed in us-west-2 sometime between May 12th and May 17th. But I know for certain the issue was fixed by May 22nd worldwide as AWS wanted to have a phone call with me to tell me they had fixed it.

On May 26th we managed to coordinate that call and they read me a verbal statement they would not provide over email or the support ticket I had with them. They told me about the mitigating circumstances and that they were fixing the issue anyway out of an abundance of caution.

I can't say for certain that I was the first one to report the issue. I just know that I reported it and the timeline in relation to my report. I reported it both via AWS Security's Vulnerability Form and and AWS Support Ticket.

I have no way of knowing what controls AWS has internally with their own employees accessing CloudTrail logs for customers. But I presume someone has some degree of access. I have no information if this was used or not to gain unauthorized access to any MWAA server.

I do not know is how long this issue was present in their systems (they didn't tell me).

If you're an AWS customer using MWAA I don't believe you need to take any specific action. All tokens that were logged like this have long since expired. But you may want to look at your logs to see if anything suspicious happened.

The verbal statement bit was rather weird. It sounded like something that had been prepared to post on a website but that they didn't want to actual post. It made me feel as though AWS wanted to hide the situation.

In my opinion AWS should publicly announce all of these vulnerabilities. Even if they are minor, difficult to exploit and even if possible to prove it was never actually used to gain unauthorized access.

I've always had the view that AWS had excellent security. But I hope everyone realizes that just because you don't hear about the issues doesn't mean they don't exist. Vendors are out there quietly fixing problems.

I will also add that the security and support engineers I exchanged messages with were great and that I am very satisfied that AWS took action on this issue as quickly as they did.