GitHub Actions: Solving Docker-in-Docker Mount Challenges in Self-Hosted Runners

The Challenge: Bind Mounts and Docker-in-Docker Isolation

A common hurdle for developers leveraging GitHub Actions self-hosted runners with containerMode: dind (Docker-in-Docker) is the unexpected behavior of bind mounts. As highlighted in recent github reports, users like 'schrom' discovered that files created within their job container were not accessible when attempting to mount them into containers launched by the dind service.

The problem manifests when a file, say /tmp/secret.txt, is created in the job container and then an attempt is made to bind mount it into a test container via docker run -v. Instead of the file, Docker either mounts an empty directory or throws an error indicating the source path does not exist. Here's 'schrom's' minimal example, which works locally but fails in the pipeline:

$ echo hello > /tmp/secret.txt
$ docker run -it -v /tmp/secret.txt:/mnt/secret.txt alpine:3
# ls -al /mnt/
total 8
drwxr-xr-x 1 root root 4096 Mar 24 17:16 .
drwxr-xr-x 1 root root 4096 Mar 24 17:16 ..
drwxr-xr-x 2 root root 40 Mar 24 17:15 secret.txt
/ # ls -al /mnt/secret.txt/
total 4
drwxr-xr-x 2 root root 40 Mar 24 17:15 .
drwxr-xr-x 1 root root 4096 Mar 24 17:16 ..

A similar issue arises with Docker Compose, leading to an Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /tmp/secret.txt.

services:
  test:
    image: alpine:3
    secrets:
      - credentials
    command: [ "cat", "/run/secrets/credentials" ]
secrets:
  credentials:
    file: "/tmp/secret.txt"

The core observation was that files were being mounted from the dind container's filesystem, not the job container's. Creating the file inside the dind sidecar itself made it accessible to the launched containers.

Understanding the "Why": DIND's Expected Behavior

As 'andreas-agouridis' clarified, this isn't a bug but expected behavior due to the isolation inherent in Docker-in-Docker setups. When containerMode: dind is used, your GitHub Actions job runs in one container, and the Docker daemon (dind) runs in a separate sidecar container. Any bind mounts specified in your workflow are relative to the *dind container's* filesystem, not the job container's.

Therefore, a file created at /tmp/secret.txt in your job container is not automatically visible or accessible within the dind container's filesystem. When the dind daemon tries to resolve the mount path, it looks within its own environment, finds nothing at /tmp/secret.txt, and thus either fails or creates an empty directory as a placeholder.

The Simple Solution: Leveraging Shared Volumes

The good news, as discovered by 'schrom' with further investigation, is that the self-hosted runner Helm chart already provides a solution: a pre-configured shared volume. This volume, mounted as /home/runner/_work, is accessible by both the runner's job container and the dind sidecar container.

The key insight is to use this shared space for any files or directories that need to be bind-mounted into containers launched by dind. GitHub Actions conveniently provides the $RUNNER_TEMP environment variable, which points to /home/runner/_work/_temp/ – a location *within* this shared volume.

By simply ensuring that generated config files, secrets, or other artifacts are placed in a directory referenced by $RUNNER_TEMP (or any other path within /home/runner/_work), they become visible to the dind container and can be successfully bind-mounted. The original mistake was hard-coding paths like /tmp/, which falls outside this shared volume.

Key Takeaways for Robust GitHub Reports

• DIND Isolation: Understand that the job container and dind sidecar have separate filesystems.
• Shared Volume is Key: Utilize the pre-existing shared volume (/home/runner/_work) provided by the self-hosted runner Helm chart.
• Use $RUNNER_TEMP: For temporary files intended for dind mounts, always use $RUNNER_TEMP to ensure they are placed within the shared volume.
• RTFM: As 'schrom' aptly put it, sometimes the simplest solution is to refer to the documentation and adhere to recommended practices.

By following these guidelines, developers can avoid common pitfalls with Docker-in-Docker setups in GitHub Actions, ensuring their build pipelines run smoothly and reliably, contributing to clearer github reports on build success.