为什么镜像可以 pull 下来但是在 manifest inspect 的时候提示 no such manifest?—— Docker Buildx Attestations 检修记

你有没有遇到过这种情况,对于一个镜像来说,它可以正常 pull 下来:

pad ~ #  docker pull knatnetwork/github-runner-amd64:focal-2.301.1
focal-2.301.1: Pulling from knatnetwork/github-runner-amd64
846c0b181fff: Pull complete 
588b3eef3b63: Pull complete 
189ea0ac146f: Pull complete 
4f4fb700ef54: Pull complete 
546945707c6e: Pull complete 
71464c2d54c9: Pull complete 
1c4efc443e6a: Pull complete 
21bbc223ea9a: Pull complete 
Digest: sha256:6b5b4aa94f8c1e781785e831d18d7ccc1a0de7d70d63b1afd4df3cce27ddd53f
Status: Downloaded newer image for knatnetwork/github-runner-amd64:focal-2.301.1
docker.io/knatnetwork/github-runner-amd64:focal-2.301.1

但是如果你想 inspect 它的 manifest 会发现 no such manifest

pad ~ # docker manifest inspect knatnetwork/github-runner-amd64:focal-2.301.1
no such manifest: docker.io/knatnetwork/github-runner-amd64:focal-2.301.1

我怎么会遇到这么个鬼问题呢?


在 2022 年 4 月,我开源了 GitHub Runner, 相关的文章是: 开源 Github Actions Self-Hosted Runner,由于这个 Runner 的 Image 就是在 GitHub Actions 上面构建的,且为了提供多架构的支持(ARM64 和 AMD64) 并为了保证构建速度,整个构建工作分为了以下几步:

  1. 第一阶段同时开两个 Runner 分别构建 knatnetwork/github-runner-amd64:focal-2.301.1knatnetwork/github-runner-arm64:focal-2.301.1 的镜像
  2. 在上面两个 Runner 完成之后通过操作 manifest 的方式合并为一个叫 knatnetwork/github-runner:focal-2.301.1 的 Multi-Arch 镜像

这么做一直没有问题,直到几天前在最后一步合并镜像的时候遇到了第一个报错: https://github.com/knatnetwork/github-runner/actions/runs/3954481625/jobs/6776296661

failed to put manifest docker.io/knatnetwork/github-runner:focal-2.301.1: errors:
manifest blob unknown: blob unknown to registry

奇怪,难道是因为 GitHub 有一些 Step 没有升级么?

想到之前看到过一堆 The set-output command is deprecated and will be disabled soon.,于是尝试升级了一下 docker/login-actiondocker/build-push-action 等,然后重新触发任务,结果依然是在合并镜像的时候报错,不过这一次报错内容还不太一样,是:

Run docker manifest create knatnetwork/github-runner:focal-2.301.1 --amend knatnetwork/github-runner-amd64:focal-2.301.1 --amend knatnetwork/github-runner-arm64:focal-2.301.1

docker.io/knatnetwork/github-runner-amd64:focal-2.301.1 is a manifest list

基于个人的经验,如果同一段代码之前能跑,现在突然不能跑了,在这个情况下,一般有如下可能:

  • GitHub Runner 环境的 Docker 版本变了
  • docker/login-actiondocker/build-push-action 中有什么变更,或者这些 step 使用的组件(比如 buildx)有啥变更
  • DockerHub/GHCR 出问题了

我们先排除最后一个可能,因为过了两天之后再重试发现问题依旧,且没有看到有大量对于这两个服务不可用的反馈,所以只剩下前两个可能。

GitHub Runner Docker

先看看是不是 Docker 有啥 Breaking change 导致的问题,最后一次成功的 Action 是: https://github.com/knatnetwork/github-runner/actions/runs/3736662591,调试信息中:

Client:
   Version:           20.10.21+azure-2
   API version:       1.41
   Go version:        go1.18.9
   Git commit:        baeda1f82a10204ec5708d5fbba130ad76cfee49
   Built:             Tue Oct 25 17:53:02 UTC 2022
   OS/Arch:           linux/amd64
   Context:           default
   Experimental:      true
  
  Server:
   Engine:
    Version:          20.10.21+azure-2
    API version:      1.41 (minimum version 1.12)
    Go version:       go1.18.9
    Git commit:       3056208812eb5e792fa99736c9167d1e10f4ab49
    Built:            Tue Oct 25 11:44:15 2022
    OS/Arch:          linux/amd64
    Experimental:     false

第一次失败开始的 Action: https://github.com/knatnetwork/github-runner/actions/runs/3954481625/jobs/6776269393 ,调试信息中:

  Client:
   Version:           20.10.22+azure-1
   API version:       1.41
   Go version:        go1.18.9
   Git commit:        3a2c30b63ab20acfcc3f3550ea756a0561655a77
   Built:             Thu Dec 15 15:37:38 UTC 2022
   OS/Arch:           linux/amd64
   Context:           default
   Experimental:      true
  
  Server:
   Engine:
    Version:          20.10.22+azure-1
    API version:      1.41 (minimum version 1.12)
    Go version:       go1.18.9
    Git commit:       42c8b314993e5eb3cc2776da0bbe41d5eb4b707b
    Built:            Thu Dec 15 22:17:04 2022
    OS/Arch:          linux/amd64
    Experimental:     false

看上去确实有一些版本升级,不过阅读了 https://docs.docker.com/engine/release-notes/#201022 之后发现基本只有点 Patch ,没有什么足以引起这种问题的更新。

那么现在压力就来到了第二个,即 「docker/login-actiondocker/build-push-action 中有什么变更,或者这些 step 使用的组件(比如 buildx)有啥变更」。

Manifest

在继续调查前我们先看一下上面的报错是个什么情况,为什么镜像能拉,但是 manifest 看不了,难道拉镜像之前不需要看 manifest 么?

Docker 用来查看 manifest 的指令是 docker manifest inspect ,但是这个指令没有类似用于调试的 -v 的选项,所以如果看到了 no such manifest,那你也没法知道背后出了啥问题,不过考虑到 manifest 就一个 JSON 文件,所以肯定是有 Docker Hub 的 API 可以查询的,于是立即上网梭了一个脚本出来:

#!/bin/sh

ref="${1:-library/ubuntu:latest}"
sha="${ref#*@}"
if [ "$sha" = "$ref" ]; then
  sha=""
fi
wosha="${ref%%@*}"
repo="${wosha%:*}"
tag="${wosha##*:}"
if [ "$tag" = "$wosha" ]; then
  tag="latest"
fi
api="application/vnd.docker.distribution.manifest.v2+json"
apil="application/vnd.docker.distribution.manifest.list.v2+json"
token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
        | jq -r '.token')
curl -H "Accept: ${api}" -H "Accept: ${apil}" \
     -H "Authorization: Bearer $token" \
     -s "https://registry-1.docker.io/v2/${repo}/manifests/${sha:-$tag}" | jq .

来源:https://stackoverflow.com/questions/57316115/get-manifest-of-a-public-docker-image-hosted-on-docker-hub-using-the-docker-regi

然后找了个正常的镜像试了一下,输出结果类似是这样的,和用 docker manifest inspect 结果一致:

{
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "digest": "sha256:19bf2d0d0a8aaf27988db772ff6ba4044405447535762bfc9ba451d0d84f0a18",
    "size": 4995
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:846c0b181fff0c667d9444f8378e8fcfa13116da8d308bf21673f7e4bea8d580",
      "size": 28576882
    },
...
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:74b36662af5e651ae3390a6cf13fcaa8fca08fea5bd711ddbed60bf9e5924654",
      "size": 932
    }
  ]
}

于是立即看了一下有问题的镜像,结果是这样的:

{
  "errors": [
    {
      "code": "MANIFEST_UNKNOWN",
      "message": "OCI index found, but accept header does not support OCI indexes"
    }
  ]
}

OCI Image Index Specification 文档中我们知道 manifest 有很多类型,大家一般在用的是 application/vnd.docker.distribution.manifest.v2+json,如果是一个 multi-arch 的镜像的话可能输出结果是这样的:

{
  "manifests": [
    {
      "digest": "sha256:93d5a28ff72d288d69b5997b8ba47396d2cbb62a72b5d87cd3351094b5d578a0",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      },
      "size": 528
    },
    {
      "digest": "sha256:176bc6c6e93528f4b729fae1f8dbd70b73861264dba3a3f64c49c92e1f42a5aa",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "s390x",
        "os": "linux"
      },
      "size": 528
    }
  ],
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "schemaVersion": 2
}

这里它的格式是 application/vnd.docker.distribution.manifest.list.v2+json ,也就是上面脚本中请求的时候同时带上了以下两个 header 的原因。

api="application/vnd.docker.distribution.manifest.v2+json"
apil="application/vnd.docker.distribution.manifest.list.v2+json"

但是这里根据提示 OCI index found ,我们猜测可能实际的 manifest 格式和上面两个都不匹配,于是加入了以下两个新的 Header 上去,显式定义一下我们还接受 application/vnd.oci.image.index.v1+json 这个格式:

api_old="application/vnd.oci.image.manifest.v1+json"
api_oldi="application/vnd.oci.image.index.v1+json"

很快,我们就看到有问题的镜像也能返回了,数据是这样的:

{
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:73809677ff2aff4bee611f1da7cdc9b8825c5729d2aab4c88b683cfa0e5fc7f0",
      "size": 1817,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:f47cf60d8b8da4e0f5040071b78ddb41f0ae160da6b1be7ddcba03a5c0bf9b3d",
      "size": 567,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:73809677ff2aff4bee611f1da7cdc9b8825c5729d2aab4c88b683cfa0e5fc7f0",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    }
  ]
}

这就很有意思了,我用:

 - name: Build and push AMD64 Version
    uses: docker/[email protected]
    with:
      context: ./amd64/
      file: ./amd64/Dockerfile
      platforms: linux/amd64
      push: true
      tags: |
    knatnetwork/github-runner-amd64:focal-${{ github.event.inputs.github-runner-version }}

构建出来的镜像为什么 manifests 是个数组(像是一个 multi-arch 的镜像),而且第二个 platform 还是 unknown?

所以应该也是这个原因导致了: docker.io/knatnetwork/github-runner-amd64:focal-2.301.1 is a manifest list 这个报错, 操作 manifest 合并镜像不能把两个多 Arch 镜像合并。

但为什么?

attestation manifest

在上文的输出中我们看到了一个关键信息: "vnd.docker.reference.type": "attestation-manifest",经过搜索看到了这个文档: Attestation storage | Docker Documentation

Buildkit supports creating and attaching attestations to build artifacts. These attestations can provide valuable information from the build process, including, but not limited to: SBOMs, SLSA Provenance, build logs, etc.

哦?是 Buildkit 搞的事情?

于是开始检查最后一次成功的 Buildx 版本,发现是:

github.com/docker/buildx 0.9.1+azure-2 ed00243a0ce2a0aee75311b06e32d33b44729689

再看看第一次失败的 Buildx 版本:

github.com/docker/buildx 0.10.0+azure-1 876462897612d36679153c3414f7689626251501

版本从 0.9.1 升级到了 0.10.0 ,这个时候回顾一下 docker/build-push-actionRelease Note 中有这么一段话:

Buildx v0.10 enables support for a minimal SLSA Provenance attestation, which requires support for OCI-compliant multi-platform images. This may introduce issues with registry and runtime support (e.g. GCR and Lambda). You can optionally disable the default provenance attestation functionality using provenance: false.

很快我们就知道这里的问题在于 Buildx 从 0.10 开始就默认加入了这个叫做 SLSA Provenance attestation 的东西,也就是我们看到的 manifest 中底下那个 "vnd.docker.reference.type": "attestation-manifest" 的内容,这么做对于直接构建的 Multi-Arch 镜像没有影响,对于单架构镜像而言一般也没有影响(虽然会在 docker manifest inspect 的时候报错),但是一旦有了像我这样多个并行构建,后期操作 manifest 的合并的操作的时候,就会导致 docker.io/knatnetwork/github-runner-amd64:focal-2.301.1 is a manifest list 类似这样的错误。

如果你想了解更多关于 Build attestations 的事情,可以从 Docker 的文档: Build attestations | Docker Documentation 开始阅读,简单来说分为 SBOM 和 Provenance:

Build attestations describe how an image was built, and what it contains. The attestations are created at build-time by BuildKit, and become attached to the final image as metadata.

Two types of build annotations are available:

  • Software Bill of Material (SBOM): list of software artifacts that an image contains, or that were used to build the image.

  • Provenance: how an image was built.

既然问题很清晰了,那解决问题的思路也明确了,在 docker/build-push-action 加入以下两行即可:

provenance: false
sbom: false

构建后我们再次通过脚本确认,发现 manifest 已经正常,如下:

{
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "digest": "sha256:82da6a4f14803932bfece329e5d2592b74dbbb65a3c493bb6b459fb8b3a082ff",
    "size": 4995
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:846c0b181fff0c667d9444f8378e8fcfa13116da8d308bf21673f7e4bea8d580",
      "size": 28576882
    },
...
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:8b5ad40966565f7a972b30cf9494aa3600645350952d99f1d442c143a03d2650",
      "size": 932
    }
  ]
}

而至于一开始遇到的 manifest blob unknown: blob unknown to registry 问题,猜测是由于合并镜像需要在一个 repo 下,逻辑应该是:

  • knatnetwork/github-runner-amd64:latestknatnetwork/github-runner-arm64:latest 不能合并
  • knatnetwork/github-runner:latest-amd64knatnetwork/github-runner:latest-arm64 可以合并

不过这里似乎也没法解释为什么之前这么做是可以的,如果有读者有了解的话,欢迎在评论区中指出。

Summary

总结,为了解决上面两个问题,我分别做了以下调整:

  1. docker/build-push-action 中显式禁用了 provenance 和 sbom 的输出
  2. 将 amd64 和 arm64 的镜像改变同 repo 的不同 tag 输出

同时得出一个结论就是:如果你和我一样想后期操作 manifest 来调整镜像的话,一定要注意 buildx 的这个新特性,要么显式禁用掉,要么考虑修改你的 Dockerfile 们尽量一次通过 buildx 构建成 Multi-Arch 的镜像。

References

  1. Attestation storage | Docker Documentation
  2. Build attestations | Docker Documentation
  3. Attestation storage | Docker Documentation
  4. Releases · docker/build-push-action · GitHub
  5. Action started to push manifest indexes instead of images for a single platform · Issue #755 · docker/build-push-action · GitHub

comments powered by Disqus