TL;DR: I coded a tool that scans Docker Hub images and matches a given keyword in order to find secrets. Using the tool, I found numerous AWS credentials, SSH private keys, databases, API keys, etc. It’s an interesting tool to add to the bug hunter / pentester arsenal, not only for the possibility of finding secrets, but for fingerprinting an organization. On the other hand, if you are a DevOps or Security Engineer, you might want to integrate the scan engine to your CI/CD for your Docker images.
GET THE TOOL: https://github.com/matiassequeira/docker_explorer
The idea for this work came up when I was opening the source code for a project on which I was collaborating. Apart from migrating the source code to an open GitHub, we prepared a ready-to-go VM that was uploaded to an S3 bucket and a few Docker images that we pushed to Docker Hub. A couple of days later, a weekend to be more specific, we got a warning from AWS stating that our SNS resources were being (ab)used – more than 75k emails had been sent in a few hours. Clearly, our AWS credentials were exposed.
Once we deleted all the potentially exposed credentials and replaced them in our environments, we started to dig into the cause of the incident and realized that the credentials were pushed to GitHub along with the source code due to a miscommunication within the team. As expected, the credentials were also leaked to the VM, but the set of Docker images were fine. Anyhow, this got me thinking about the possibility of scanning an entire Docker images repository, the same way hackers do with source code repositories. Before starting to code something, I had to check whether it was possible.
Analyzing Feasibility
Getting a list of images
The first thing I had to check was if it was possible to retrieve a list of images that match a specific keyword. By taking a look at the API URLs using a proxy, I found:
https://hub.docker.com/v2/search/repositories?query={target}&page={page}&page_size={page_size}The only limitation I found with API V2 was that it wouldn’t retrieve anything beyond page number 100. So, given the maximum page size of 100, I wouldn’t be able to scan more than 10k containers per keyword. This API also allows you to sort the list by pull count, so, if we retrieve the repositories with fewer pulls (in other words, the newest repositories), we have a greater chance of finding something fresh. Although there’s a V1 of the API that has many other interesting filters, it wouldn’t allow me to retrieve more than ~2.5k images.
After getting the image name, and since not all the images had the `latest` version, I had to make a second request to get a list of versions for each image to the following endpoint:
https://hub.docker.com:443/v2/repositories/{image}/tags/?page_size=25&page=1Once I had the `image:version`, the only thing left was to pull the image, create a temporary container, and dump its filesystem.
Analyzing the image dump
This was one of the most important parts of my research, because it involved the engine I used for scanning the images. Before trying to create a scan engine, which is a big enough problem, and with so many options to choose from, I started to look for the most suitable tool which could work on an entire filesystem, which could look for a wide variety of secrets. After doing some research, I found many options, but sadly, most of them were oriented to GitHub secrets, had a high rate of false positives, or evaluated secrets as isolated strings (without considering format var=value).
While discussing this with one of my colleagues, he mentioned a tool that had been published less than a week ago called Whispers, a great tool by Artëm Tsvetkov and Christian Martorella. By doing some tests, I found this tool very convenient for several reasons:
- Contains many search rules (AWS secrets, GitHub secrets, etc.) that assess potential secrets as var=value format
- Findings are classified by type, impact (minor, major, critical, blocker), file location, etc.
- Allows you to add more rules and plugins
- Written in Python3 and thus very easy to modify
Developing the tool
Once I had everything in place, I started to work on a script to automate the process in order to scan thousands of images. By using/testing the tool, I came up with additional requirements, such as allowing the user to provide the number of core processors to use, limit Whispers execution time, store logs separately for each container, delete containers and images in order to avoid filling up the disk space, etc.
Also, in order to maximize the number of findings, minimize the number of false positives, and ease data triage, I made a couple of modifications to the standard Whispers:
- Added a rule for Azure stuff
- Excluded many directories and files
- Saved files with potential secrets into directories
Running the tool
With the tool pretty much ready to analyze bulk images, I signed up for two different DigitalOcean accounts and claimed $100 in credit for each. Later, I spun up two servers, set up the tool in each environment, and ran the tool using a large set of keywords/targets.