panic: invalid OCI Image Index: trailing data

[ahbarrios]

I found this recently in my production code using opa SDK in go. The panic is been triggered here but caused more precisely for this code https://github.com/oras-project/oras-go/blob/main/content/oci/oci.go#L116 per error prefix. Here is the function under analysis https://github.com/oras-project/oras-go/blob/main/content/oci/oci.go#L370-L395 but all I can say so far is that nothing looks bad in the index.json other than I noticed that the actual Manifest size say 0 when this panics. But no obvious reason behind that.

I tried to debug calling this same folder structure with the oci.New function but then I get a different error.

package main

import (
        "flag"
        "fmt"
        "runtime/debug"

        "oras.land/oras-go/v2/content/oci"
)

var (
        storePath *string
)

func init() {
        storePath = flag.String("path", "/tmp/opa/oci", "path to store")
        flag.Parse()
}

func main() {
        if _, err := oci.New(*storePath); err != nil {
                fmt.Printf("error: %v\nstack trace:\n%s", err, debug.Stack())
        }
}

Here is an index.json that is failing atm:

{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:5888201c59fcd90a80861345c2269ae10c2915ede85947681829c73f8191f46c",
      "size": 0,
      "annotations": {
        "org.opencontainers.image.ref.name": "company/services/example:latest"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:6b9e88db3094af75fb5ccd5b0344fb7ff4250423061f91681172240d6e5dc80d",
      "size": 573,
      "annotations": {
        "org.opencontainers.image.ref.name": "comany/services/settings:latest"
      }
    }
  ]
}

Can anyone throw some light about this issue or point me into the right direction please to debug this in my actual codebase..

[Wwwsylvia]

Hi @ahbarrios, I tried to reproduce the issue using the index.json you provided and got the following error:

invalid OCI Image Index: sha256:5888201c59fcd90a80861345c2269ae10c2915ede85947681829c73f8191f46c: application/vnd.oci.image.manifest.v1+json: invalid digest

Is this the error message you saw?

It looks like sha256:5888201c59fcd90a80861345c2269ae10c2915ede85947681829c73f8191f46c is not a valid digest. How did this index.json get produced? 🤔

[ahbarrios]

Yess I think so

[ahbarrios]

For what I can see the issue was caused initially, at least on my env due to a concurrency issue while reading the bundle from OPA server at the same time it was write. I'm assuming this happens because the file was truncated at the time of reading and the OPA SDK read a truncated 0 value. Even though that bundle manifest was properly retrieved. What is odd to me is what unrelated issue this looks like. I reviewed the index.json been generated and it doesn't look like it has any trailing data. So this distract me for a while from the real problem.

[Wwwsylvia]

It sounds like this issue can be fixed by regenerating the index.json?

[ahbarrios]

I think this is related somehow #286 the concurrency happens at write time but not triggering by helm if more the client use case whenever you use OPA SDK with an OCI bundle. Which use this library under the hood. I was hable to repro easily trying to benchmark sdk.New with an OCI server. The index creation isn't thread safe at all. I dunno if this should be addressed by the client of this library or this is something to take into account here. At least in documentation.

[Wwwsylvia]

Yeah the index.json is not protected across process, it's only locked within the same store instance. Do you mean that the broken index was produced by oras-go v2 somehow?