Object Store "Get" API call always fails to verify content digest

[domderen]

Defect

Hey, I'm working on re-writing the Object Store client implementation from Golang to Python, and I think I found a bug in the golang's implementation of ObjectStore.Get API. Near the end of this API call, there is a code that performs a digest check between the content it just obtained from NATS, and the digest saved in ObjectInfo instance:

if tokens[ackNumPendingTokenPos] == objNoPending {
	pw.Close()
	m.Sub.Unsubscribe()

	// Make sure the digest matches.
	sha := h.Sum(nil)
	rsha, err := base64.URLEncoding.DecodeString(info.Digest)
	if err != nil {
		gotErr(m, err)
		return
	}
	if !bytes.Equal(sha[:], rsha) {
		gotErr(m, ErrDigestMismatch)
		return
	}
}

But the value of info.Digest is equal to something like sha-256=X7IFRHg1P9jVFAVtF0Wzqe7wZt6t2kuQlnr3ymXOZQU= because it was created by this line in the ObjectStore.Put API call:

info.Digest = fmt.Sprintf(objDigestTmpl, base64.URLEncoding.EncodeToString(sha[:]))

Because of that, the digest validation always fails in the Get API call, because the string that gets decoded into an SHA is not a valid Base64 value.

But what is worse, it looks like this error is not even propagated correctly into the calling code because of a race condition. I created a small sample that visualizes this problem and attached it below. I had to modify the Get API call with some additional print statements to visualize this issue.

Could someone confirm if that is indeed an issue, or if I'm misunderstanding something about how it should be used?

Thanks in advance!

Versions of nats.go and the nats-server if one was involved:

github.com/nats-io/nats.go v1.16.0
NATS-Server version 2.8.2

OS/Container environment:

Tested on MacOS, but I think this error occurs on all environments.

Steps or code to reproduce the issue:

Small code sample that presents the bug:

package main

import (
	"fmt"
	"io/ioutil"
	"time"

	"github.com/nats-io/nats.go"
)

func main() {
	nc, err := nats.Connect("nats://localhost:4222")
	if err != nil {
		fmt.Println(err)
		return
	}
	defer nc.Close()

	js, err := nc.JetStream()
	if err != nil {
		fmt.Println(err)
		return
	}

	obs, err := js.ObjectStore("something")
	if err != nil {
		fmt.Println(err)
		return
	}

	_, err = obs.PutString("sometext", "sometext")
	if err != nil {
		fmt.Println(err)
		return
	}

	res, err := obs.Get("sometext")
	if err != nil {
		fmt.Println(err)
		return
	}

	fmt.Println("RES ERR", res.Error())

	buf, err := ioutil.ReadAll(res)
	if err != nil {
		fmt.Println(err)
		return
	}

	fmt.Println("RES ERR2", res.Error())

	fmt.Println("OUTPUT TEXT: ", string(buf[0:10]))
	for {
		time.Sleep(10 * time.Second)
	}
}

I also modified my local version of func (obs *obs) Get(name string, opts ...ObjectOpt) (ObjectResult, error) function with additional prints lines to better visualize the problem:

// Get will pull the object from the underlying stream.
func (obs *obs) Get(name string, opts ...ObjectOpt) (ObjectResult, error) {
	// Grab meta info.
	info, err := obs.GetInfo(name)
	if err != nil {
		return nil, err
	}
	if info.NUID == _EMPTY_ {
		return nil, ErrBadObjectMeta
	}

	// Check for object links.If single objects we do a pass through.
	if info.isLink() {
		if info.ObjectMeta.Opts.Link.Name == _EMPTY_ {
			return nil, errors.New("nats: link is a bucket")
		}
		lobs, err := obs.js.ObjectStore(info.ObjectMeta.Opts.Link.Bucket)
		if err != nil {
			return nil, err
		}
		return lobs.Get(info.ObjectMeta.Opts.Link.Name)
	}

	var o objOpts
	for _, opt := range opts {
		if opt != nil {
			if err := opt.configureObject(&o); err != nil {
				return nil, err
			}
		}
	}
	ctx := o.ctx

	result := &objResult{info: info, ctx: ctx}
	if info.Size == 0 {
		return result, nil
	}

	pr, pw := net.Pipe()
	result.r = pr

	gotErr := func(m *Msg, err error) {
		pw.Close()
		m.Sub.Unsubscribe()
		result.setErr(err)
	}

	// For calculating sum256
	h := sha256.New()

	processChunk := func(m *Msg) {
		if ctx != nil {
			select {
			case <-ctx.Done():
				if ctx.Err() == context.Canceled {
					err = ctx.Err()
				} else {
					err = ErrTimeout
				}
			default:
			}
			if err != nil {
				gotErr(m, err)
				return
			}
		}

		tokens, err := getMetadataFields(m.Reply)
		if err != nil {
			gotErr(m, err)
			return
		}

		// Write to our pipe.
		for b := m.Data; len(b) > 0; {
			n, err := pw.Write(b)
			if err != nil {
				gotErr(m, err)
				return
			}
			b = b[n:]
		}
		// Update sha256
		h.Write(m.Data)

		// Check if we are done.
		if tokens[ackNumPendingTokenPos] == objNoPending {
			fmt.Println("call1")
			pw.Close()
			fmt.Println("call2")
			m.Sub.Unsubscribe()
			fmt.Println("call3")

			// Make sure the digest matches.
			sha := h.Sum(nil)
			fmt.Println("call4", sha, info.Digest)
			rsha, err := base64.URLEncoding.DecodeString(info.Digest)
			fmt.Println("call5", rsha)
			if err != nil {
				fmt.Println("call6!", err)
				gotErr(m, err)
				return
			}
			fmt.Println("call7", err)
			if !bytes.Equal(sha[:], rsha) {
				gotErr(m, ErrDigestMismatch)
				return
			}
		}
	}

	chunkSubj := fmt.Sprintf(objChunksPreTmpl, obs.name, info.NUID)
	_, err = obs.js.Subscribe(chunkSubj, processChunk, OrderedConsumer())
	if err != nil {
		return nil, err
	}

	return result, nil
}

This produced the following output in my console:

RES ERR <nil>
call1
call2
RES ERR2 <nil>
OUTPUT TEXT:  sometext
call3
call4 [95 178 5 68 120 53 63 216 213 20 5 109 23 69 179 169 238 240 102 222 173 218 75 144 150 122 247 202 101 206 101 5] sha-256=X7IFRHg1P9jVFAVtF0Wzqe7wZt6t2kuQlnr3ymXOZQU=
call5 [178 22 190 219 158]
call6! illegal base64 data at input byte 8

As you can see the call7 never got called, even though I was just putting and getting an object from the ObjectStore. Also, print statements "RES ERR" & "RES ERR2" both times printed indicating that there was no error while getting object from the object store, even though call6! print statement indicates otherwise.

[derekcollison]

Will take a look, thanks!

[scottf]

info.digest is actually sha-256=X7IFRHg1P9jVFAVtF0Wzqe7wZt6t2kuQlnr3ymXOZQU= We should decode only the part after sha-256=