What a mysterious bug taught us about how Docker stores registry credentials
We recently ran into a mysterious bug that required hours of digging into the arcane details of Docker’s registry credentials store to figure out. Although in the end the fix turned out to be easy, we learned a thing or two along the way about the design of the credentials store and how, if you’re not careful, it can be configured insecurely.
Blimp, sometimes needs to pull private images from a Docker registry in order to boot those images in the cloud. This typically works fine, but unfortunately, when some users started Blimp, they were getting the following error message:
Get https://1234.dkr.ecr.us-east-1.amazonaws.com/v2/blimp/blimp/manifests/v0.1: no basic auth credentialsAt first, we were completely baffled by this cryptic message and had no clue it was related to our handling of credentials. To understand how we figured it out, first you need to know a little about how modern Docker credentials are handled.
Docker’s External Credentials Store
The recommended way to store your Docker credentials is in an external credentials store. In your Docker config file, which is usually located at ~/.docker/config.json, there are two fields you can use to configure how Docker gets and stores credentials: credsStore and credHelpers.
credsStore tells Docker which helper program to use to interact with the credentials store. All helper programs have names that begin with docker-credential- – the value of credsStore is the suffix of the helper program.
For example, if you work on a Mac laptop, you might decide to use the Mac OS keychain. The name of the helper program to use the keychain is docker-credential-osxkeychain. So your config.json would include the following:
{
"credsStore": "osxkeychain"
}If you want to see what credentials Docker currently has for you, you can use list. For example:
docker-credential-osxkeychain listThe result is a list of pairs of servers and usernames. For example:
{
"http://quay.io":"kklin",
"https://index.docker.io":"kevinklin"
}You may also notice credHelpers in your config.json. These helpers are similar to credsStore, but are used to generate short lived credentials. For example, if you use gcr, gcloud installs a credHelper that uses your Google login to get tokens. This way, Docker never has your Google credentials directly – the docker-credential-gcloud acts as a middleman between Docker and your Google credentials.
Once again, here’s the error message our users were getting:
Get https://1234.dkr.ecr.us-east-1.amazonaws.com/v2/blimp/blimp/manifests/v0.1: no basic auth credentialsWe were able to run the docker-credential-osxkeychain list and get commands to see the credentials for 1234.dkr.ecr.us-east-1.amazonaws.com, so why were we getting an error that there weren’t any credentials??
In the Beginning: Docker Stores Your Registry Password In Your Config File
It turns out that external credentials stores weren’t added to Docker until version 1.11, in 2016. Before 1.11, Docker stored credentials via a config field called auths. This field is stored in the same file as the credStore: ~/.docker/config.json.
Whenever you logged into a registry, Docker would set the value of auths to your password. For example, your config file might contain the following:
{
"auths": {
"https://index.docker.io/v1/": {
"auth": "YW11cmRhY2E6c3VwZXJzZWNyZXRwYXNzd29yZA=="
},
"localhost:5001": {
"auth": "aGVzdHVzZXI6dGVzdHBhc3N3b3Jk"
}
}
}What we learned the hard way is that there’s a quirk with Docker’s login command. When you log in using docker login, Docker adds an entry via the credsStore and in auths, using slightly different server names. Your credentials are properly stored in the credentials store, but the entry in auths doesn’t contain the username or password. The result looks something like this:
{
"auths": {
"https://index.docker.io/v1/": {}
}The problem is that Blimp grabs credentials from both auths and credsStore. So it was passing two copies of the credentials to the Docker image puller – one with the correct username and password, and one without the password at all.
Unfortunately, Docker preferred the https:// version of the credential, and attempt to pull the image with the empty credential. Thus, the no basic auth credentials error.
Once we figured out that the problem was that an empty duplicate entry was getting added to the insecure store, it was easy to fix the problem. All we needed to do was add an if statement to skip empty credentials:
addCredentials := func(authConfigs map[string]clitypes.AuthConfig) {
for host, cred := range authConfigs {
// Don't add empty config sections.
if cred.Username != "" ||
cred.Password != "" ||
cred.Auth != "" ||
cred.Email != "" ||
cred.IdentityToken != "" ||
cred.RegistryToken != "" {
creds[host] = types.AuthConfig{
Username: cred.Username,
Password: cred.Password,
Auth: cred.Auth,
Email: cred.Email,
ServerAddress: cred.ServerAddress,
IdentityToken: cred.IdentityToken,
RegistryToken: cred.RegistryToken,
}
}
}
}A Potential Docker Credentials Security Risk
In the process of uncovering this bug, we noticed a potential security risk that you may not be aware of. As we learned, it’s best practice to use an external store to store your external registry credentials. However, depending on how and when you installed Docker it’s possible you could still be using the legacy auths method. If you are, your ~/.docker/config.json might look something like this:
{
"auths": {
"https://index.docker.io/v1/": {
"auth": "YW11cmRhY2E6c3VwZXJzZWNyZXRwYXNzd29yZA=="
},
"localhost:5001": {
"auth": "aGVzdHVzZXI6dGVzdHBhc3N3b3Jk"
}
}
}This may look reasonable secure, the passwords appear to be a garbled bunch of gibberish. Surely those passwords are encrypted, right?
Guess again. All Docker did was encode the passwords using base64. And as David Rieger pointed out on Hacker Noon, base64
may look like encryption on first glance, but it’s not. Base64 is a scheme for encoding, not encryption. You can simply copy the base64 string and convert it to ASCII in a matter of seconds.
That seemingly secure password of aGVzdHVzZXI6dGVzdHBhc3N3b3Jk? All you need to do to read the password is base64 decode it:
$ echo aGVzdHVzZXI6dGVzdHBhc3N3b3Jk| base64 -D
hestuser:testpasswordThe Moral of Our Story: Double Check Your Docker Credentials’ Security
So that’s the bad news: if Docker config file isn’t properly set up, Docker is storing your credentials password in plain text.
The good news is that it’s easy to fix the problem.
All you and your team members need to do is take a quick look at ~/.docker/config.json. If it contains an auths password, get rid of it and switch over to using a credentials store. To do so, just download the appropriate docker-credential- helper for your system, and update the credsHelper field in ~/.docker/config.json.
Hope that helps!
