Integrate Concourse with Vault for Credential Management

When it comes to credential management, two names come to mind: Hashicorp Vault and Cloud Foundry CredHub. There are plenty of tutorials and guides for both of them since Concourse is from Cloud Foundry (Pivotal Cloud Foundry to be exact) as well, integration of CredHub is well documented.

I decided to give Vault a try and find that even though there are plenty of guides around, most of them omit the pre-assumption of how Concourse is deployed. With different methods of deploying Concourse, there’s nuance to the Vault integration.

This article will explore deploying Concourse and Vault v0.9.6 (plus Consul v1.0.6 as service discovery and KV store backend) using docker containers separately (alternatively you can put both into the same docker compose file and life will become so much easier).

Deploy Vault and Consul Docker container

You can either:

Manually set up vault and consul docker container

follow the instructions on Consul official website

docker pull consul
docker exec -it consul

# configure Consul

follow the instructions on Vault official website

docker pull vault
docker exec -it vault

# Configure Vault

Or

use “ cault “ (thanks to @tolitius!)

git clone git@github.com:tolitius/cault.git
# write "policy.hcl" with following content
# and put it into config/vault/policies folder
path "concourse/*" {
 policy = "read"
 capabilities = ["read", "list"]
}

# spin up containers
docker-compose up -d

# connect to vault docker container
docker exec -it cault_vault_1 sh

# check vault status
vault status

# initialize vault
vault operator init

# record unsealing key and root token

# unseal vault with 3 unsealing key
vault unseal (3 times)

# authorize vault
vault auth

# back out of vault container
exit

# use vault command without logging into vault container
alias vault='docker exec -it cault_vault_1 vault "$@"'

# write concourse policy to Vault
vault policy write policy-concourse /policies/policy.hcl

# create a renewable token for Concourse atc to use
vault token create --policy=policy-concourse -period="24h" -format=json -tls-skip-verify -renewable

# The token will look like this
{
 "request_id": "01f16efe-1ff7-7898-4101-2e257e2d6223",
 "lease_id": "",
 "lease_duration": 0,
 "renewable": false,
 "data": null,
 "warnings": null,
 "auth": {
 "client_token": "",
 "accessor": "",
 "policies": [
 "default",
 "policy-concourse"
 ],
 "metadata": null,
 "lease_duration": 86400,
 "renewable": true
 }
}

# Write down the client_token, we'll need it for Concourse deployment

DEPLOY CONCOURSE DOCKER CONTAINER

git clone https://github.com/concourse/concourse-docker

# find out vault container IP address
docker exec -it cault_vault_1 sh
ip a => it should be 172.xx.xx.xx

# modify the docker-compose.yml for concourse
# for concourse web node, change command to
command: web --vault-insecure-skip-verify
# add vault environment variables to web node
CONCOURSE_VAULT_URL: http://:8200
CONCOURSE_VAULT_PATH_PREFIX: /concourse
CONCOURSE_VAULT_CLIENT_TOKEN: ${vault_client_token}
CONCOURSE_VAULT_INSECURE_SKIP_VERIFY: "true"

# deploy concourse server
docker-compose -f docker-compose.yml up -d

Container to Container Networking

Now Concourse server still can’t connect Vault server because they are not in the same network (you’ll see atc.credential_manager.renew connection refused failure in concourse web node log), we resolve this issue by using docker container bridge networking

# find out the network Concourse nodes are using
docker network ls

# user project specific network has "_default" after the project folder name by default
# add the vault container to concourse container networking
docker network connect

To be certain, log into Concourse docker container and do a curl on Vault container IP address and port to make sure the connection is not refused

docker exec -it  bash
curl http://:8200

Write Secrets to Vault

Last, write some secret to Vault and have a test pipeline ready to verify Concourse can read the secret from Vault, a caveat here is the secret needs to written in a path contains team name determined by Concourse look up rule

/concourse/TEAM_NAME/PIPELINE_NAME/foo_param
or
/concourse/TEAM_NAME/foo_param

From host server

Enable secrets

vault secrets enable -path=/concourse -description="Secrets for use by concourse pipelines" generic

Write Secrets

vault write -address=http://127.0.0.1:8200 concourse///username value=concourse

Read it back from Concourse pipeline

---
jobs:
- name: smoketest
  plan:
  - task: say-hello
  config:
    platform: linux
    image_resource:
      type: docker-image
      source:
        repository: alpine
        tag: latest
    run:
    path: sh
    args:
    - -exc
    - |
      echo ${USERNAME}
    params:
    USERNAME: ((username))

Optional:

To check from Concourse you can indeed fetch credentials, we can use Vault HTTP API:

# Log into Concourse docker container
docker exec -it  bash

# GET secret
curl -H "X-Vault-Token: " -X GET http://:8200/v1/concourse///username

# LIST secrets
curl -H "X-Vault-Token:" -X LIST http://:8200/v1/concourse/

I’m working on bring all these manual steps above under the same umbrella, stay tuned 🙂

References:

Docker compose for consul and vault official images — Cault

Integrate Vault with Concourse using BOSH

Another example of deploy Concourse with Vault using BOSH

Concourse Docker Configurator

Concourse Docker Boiler Plate