Troubleshooting¶

Error response from daemon: manifest for ghcr.io/jinalshah/devops/images/all-devops:1.0 not found

1. Check the path. The three registries use different paths:

   ghcr.io/jinalshah/devops/images/all-devops:latest
   registry.gitlab.com/jinal-shah/devops/images/all-devops:latest
   js01/all-devops:latest
   

2. Check the tag. Only latest, 1.0.<7-char sha> and 1.0.<sha>-amd64 / -arm64 exist. There is no 1.0 tag and no semver tags. To list published tags on GHCR (the packages API needs authentication even for public images, and your gh token needs the read:packages scope: gh auth refresh -s read:packages):

   gh api /users/jinalshah/packages/container/devops%2Fimages%2Fall-devops/versions \
     --jq '.[].metadata.container.tags[]' | head -20
   

3. Try another registry if one is having an outage:

   docker pull registry.gitlab.com/jinal-shah/devops/images/all-devops:latest
   

The images are public, so you shouldn't need docker login to pull. If you are logged in with an expired token, docker logout ghcr.io and try again.

Docker Hub throttles anonymous pulls. Either:

• pull from GHCR instead: docker pull ghcr.io/jinalshah/devops/images/all-devops:latest, or

• log in to Docker Hub (docker login) so the higher authenticated limit applies. In GitHub Actions:

  - uses: docker/login-action@v4
    with:
      username: ${{ secrets.DOCKERHUB_USERNAME }}
      password: ${{ secrets.DOCKERHUB_TOKEN }}
  

The images are about 1.5 to 1.6 GB compressed, so a first pull takes a while. In CI, retry transient failures:

for i in 1 2 3; do
  docker pull ghcr.io/jinalshah/devops/images/all-devops:latest && break
  echo "Attempt $i failed, retrying..."
  sleep $((i * 10))
done

Pick the smallest image that has what you need (aws-devops or gcp-devops), and on self-hosted runners keep the image cached between jobs.

The images are published for linux/amd64 and linux/arm64, and Docker normally picks the right one. If you get the wrong one (for example after pulling a -amd64 tag on Apple Silicon):

# What did you get?
docker run --rm ghcr.io/jinalshah/devops/images/all-devops:latest uname -m
# x86_64 = amd64, aarch64 = arm64

# Force the native platform
docker pull --platform linux/arm64 ghcr.io/jinalshah/devops/images/all-devops:latest

Use the plain 1.0.<sha> tag (the multi-arch manifest) rather than an arch-suffixed one unless you really need a specific architecture.

The container runs as root, so files it writes to a bind mount belong to root on Linux hosts.

Fix ownership in the same run:

docker run --rm -v "$PWD":/srv -w /srv \
  ghcr.io/jinalshah/devops/images/all-devops:latest \
  sh -c 'terraform fmt -recursive && chown -R '"$(id -u):$(id -g)"' /srv'

Docker Desktop on macOS and Windows maps ownership for you, so this mostly affects Linux hosts.

zsh: permission denied: ./script.sh

• Make the script executable on the host (chmod +x script.sh), or run it with bash script.sh.

• On SELinux hosts (Fedora, RHEL, Rocky), relabel the mount with :z (shared) or :Z (private):

  docker run --rm -v "$PWD":/srv:z -w /srv \
    ghcr.io/jinalshah/devops/images/all-devops:latest ls
  

The default command is /bin/zsh, which exits straight away without a terminal. Use -it for an interactive shell, or give it a command:

docker run -it --rm ghcr.io/jinalshah/devops/images/all-devops:latest

# Keep a container running in the background, then exec into it
docker run -d --name devops ghcr.io/jinalshah/devops/images/all-devops:latest sleep infinity
docker exec -it devops zsh

Each image is about 4.6 to 5 GB unpacked, and old tags add up quickly.

docker system df            # what is using space
docker image prune -a       # remove unused images
docker system prune         # remove stopped containers, networks and dangling images

On Docker Desktop, you can also raise the disk limit in Settings → Resources.

Aliases are defined in ~/.zshrc and ~/.bashrc, so they only exist in interactive Zsh and Bash shells. They aren't available in docker run <image> <command>, CI steps, Fish, or when you override HOME. Use the full command there. See the alias list.

Only aws-devops and all-devops include the AWS CLI. Give the container credentials in one of these ways:

docker run --rm -v ~/.aws:/root/.aws \
  -e AWS_PROFILE=my-profile \
  ghcr.io/jinalshah/devops/images/aws-devops:latest \
  aws sts get-caller-identity

docker run --rm \
  -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN \
  -e AWS_REGION=eu-west-2 \
  ghcr.io/jinalshah/devops/images/aws-devops:latest \
  aws sts get-caller-identity

Only gcp-devops and all-devops include gcloud.

docker run --rm -v ~/.config/gcloud:/root/.config/gcloud \
  ghcr.io/jinalshah/devops/images/gcp-devops:latest \
  gcloud auth list

docker run --rm -v /path/to/sa.json:/secrets/sa.json:ro \
  ghcr.io/jinalshah/devops/images/gcp-devops:latest \
  sh -c 'gcloud auth activate-service-account --key-file=/secrets/sa.json && gcloud projects list'

gcloud auth login                         # for gcloud commands
gcloud auth application-default login     # for Terraform and client libraries

Mount your SSH directory read-only:

docker run -it --rm -v ~/.ssh:/root/.ssh:ro -v "$PWD":/srv -w /srv \
  ghcr.io/jinalshah/devops/images/all-devops:latest

If SSH complains about key permissions, fix them on the host (chmod 600 ~/.ssh/id_ed25519). If your key lives in an agent (for example 1Password or macOS Keychain) rather than a file, use HTTPS with gh auth login instead.

You're probably in the wrong image. aws and session-manager-plugin are only in all-devops aws-devops, and gcloud, gsutil and bq are only in all-devops gcp-devops.

Some tools aren't in any image: the Docker CLI, standalone kustomize (use kubectl apply -k), yq, terraform-docs and gcloud alpha. The tool reference lists what's included.

kubectl has no kubeconfig. Mount yours, or generate one inside the container:

# Mount your host kubeconfig
docker run -it --rm -v ~/.kube:/root/.kube \
  ghcr.io/jinalshah/devops/images/all-devops:latest

# Or point at a specific file
docker run -it --rm -v ~/.kube/prod.yaml:/kubeconfig:ro -e KUBECONFIG=/kubeconfig \
  ghcr.io/jinalshah/devops/images/all-devops:latest

# EKS
aws eks update-kubeconfig --name my-cluster --region eu-west-2

If your kubeconfig points at 127.0.0.1 (kind, minikube, Docker Desktop), that address means the container itself. Use --network host on Linux, or host.docker.internal on Docker Desktop.

gke-gcloud-auth-plugin is installed in all-devops and gcp-devops, so GKE works without extra setup. If it still fails:

# 1. Is gcloud authenticated, and on the right project?
gcloud auth list
gcloud config get-value project

# 2. Use the cluster's real location: --region for regional clusters, --zone for zonal ones
gcloud container clusters list
gcloud container clusters get-credentials my-cluster --region europe-west2 --project my-project

# 3. Is the plugin there?
gke-gcloud-auth-plugin --version

• Permission denied or 403: your account needs at least roles/container.clusterViewer to fetch credentials, plus Kubernetes RBAC rights for what you then run.
• executable gke-gcloud-auth-plugin not found: you're using a kubeconfig written on another machine or in an older image. Run get-credentials again inside this container.
• Private clusters: the container needs network access to the control-plane endpoint (VPN, authorised networks or DNS-based endpoint).

Another run holds the lock, or a crashed run left it behind. Make sure nothing else is running, then:

terraform force-unlock <LOCK_ID>

For S3 backends, use_lockfile = true is the current locking option; dynamodb_table is deprecated. Check that the container has credentials for the backend (see Authentication).

No vulnerability database is baked into the image, so each fresh container downloads it. Cache it between runs:

docker run --rm -v ~/.cache/trivy:/root/.cache/trivy -v "$PWD":/srv -w /srv \
  ghcr.io/jinalshah/devops/images/all-devops:latest trivy fs .

If the cache is corrupt, reset it with trivy clean --all. Behind a proxy, pass HTTPS_PROXY into the container.

A bare pip may belong to the distribution's Python rather than the default Python 3.14. Always use:

python3 -m pip install <package>

Your project isn't mounted, or the working directory isn't set. Use the standard mount:

docker run --rm -v "$PWD":/srv -w /srv -v ~/.ssh:/root/.ssh:ro \
  ghcr.io/jinalshah/devops/images/all-devops:latest \
  ansible-playbook -i inventory.ini playbook.yml

• Interactive: run claude, then /login. claude auth status shows the current state.
• CI or scripts: set ANTHROPIC_API_KEY, or CLAUDE_CODE_OAUTH_TOKEN (created with claude setup-token, which needs a Claude subscription). There is no CLAUDE_API_KEY.
• Persist it: mount ~/.claude, and also ~/.claude.json if you want the global state.
• Script hangs or prints a UI: add -p. Without it, claude starts the interactive interface even when output is redirected.

codex login                  # ChatGPT sign-in
codex login --device-auth    # no browser in the container
printenv OPENAI_API_KEY | codex login --with-api-key
codex login status

In CI, set CODEX_API_KEY and run codex exec "...". Credentials live in ~/.codex/auth.json, so mount ~/.codex and treat it like a password.

• Interactive: run copilot, then /login (a device-code flow that works without a browser).
• Token: set COPILOT_GITHUB_TOKEN (or GH_TOKEN / GITHUB_TOKEN) to a fine-grained personal access token with the "Copilot Requests" permission. Classic ghp_ tokens are rejected.
• You need an active Copilot subscription.
• Non-interactive runs need --allow-all-tools: copilot -p "..." --allow-all-tools.
• The image has the standalone copilot CLI, not the old gh copilot extension.

Antigravity CLI replaced Google's Gemini CLI, and the gemini command isn't in the image.

• Interactive: run agy. With no browser it prints a URL; open it on your machine, sign in with Google, and paste the code back. Use /login and /logout inside agy. There's no agy login subcommand.
• Persist it: mount ~/.gemini. In containers there's no keyring, so tokens are stored in files there.

• Gemini API key (headless): exporting GEMINI_API_KEY alone isn't enough. Also create ~/.gemini/antigravity-cli/settings.json containing:

  {"modelProvider": "gemini"}
  

• Application Default Credentials: run gcloud auth application-default login (or set GOOGLE_APPLICATION_CREDENTIALS), then export AGY_ADC_AUTH=true.

• Headless failures exit with code 3 and print AGY_ERROR: {...} on stderr, which usually names the problem.

ERROR: failed to solve: process "/bin/sh -c ..." did not complete successfully

Most failures are transient upstream downloads, so retry first. Then:

# Force fresh downloads
docker build --no-cache --target all-devops -t all-devops:local .

# Behind a corporate proxy
docker build \
  --build-arg HTTP_PROXY=http://proxy:8080 \
  --build-arg HTTPS_PROXY=http://proxy:8080 \
  --target all-devops -t all-devops:local .

• Make sure the version exists on the tool's release page, and pass it without a leading v (for example 0.68.14, not v0.68.14).

• Change one argument at a time:

  docker build --build-arg TERRAGRUNT_VERSION=0.68.14 \
    --target all-devops -t all-devops:test .
  

• Python needs both arguments, a full version and the matching binary name:

  docker build \
    --build-arg PYTHON_VERSION=3.13.7 \
    --build-arg PYTHON_VERSION_TO_USE=python3.13 \
    --target all-devops -t all-devops:py313 .
  

Compiling Python with optimisations is the heaviest step. Give Docker more memory (Docker Desktop: Settings → Resources, 8 GB or more is comfortable), and build the shared base once so later targets reuse its cache:

docker build --target base -t devops-base:local .
docker build --target aws-devops -t aws-devops:local .

CI builds each architecture natively. Locally, cross-building works through emulation but is slow:

docker buildx create --name multiarch --use

# Load a single platform into your local image store
docker buildx build --platform linux/arm64 --target all-devops -t all-devops:arm64 --load .

Multi-platform builds (--platform linux/amd64,linux/arm64) can't be loaded into the classic image store, so push them to a registry with --push instead. The containerd image store (the default in Docker Desktop and for new installs of Docker Engine 29 and later) can load them.

python3 -m pip install --upgrade zensical
zensical serve

Zensical is also pre-installed in every image, so you can preview the docs from a container:

docker run --rm -it -p 8000:8000 -v "$PWD":/srv -w /srv \
  ghcr.io/jinalshah/devops/images/all-devops:latest \
  zensical serve -a 0.0.0.0:8000

Stop the server with Ctrl+C, then rebuild from scratch and serve again:

zensical build --clean
zensical serve

If the page still looks stale, hard-refresh the browser.