Unable to read objects in GCS after a first request with GS_NO_SIGN_REQUEST

[arredond]

What is the bug?

Trying to read files from Google Cloud Storage with auth config options (like GS_OAUTH2_PRIVATE_KEY and GS_OAUTH2_CLIENT_EMAIL) always fails if a previous unauthenticated request (with GS_NO_SIGN_REQUEST=YES) was made.

For context, I want to first check if they can be publicly accessed and, if not, follow up with a signed request. While the signed requests on their own work fine, they always fail in the above stated case. No amount of other config options (like using GS_NO_SIGN_REQUEST=NO or trying to disable the VSI cache) fix the issue.

Steps to reproduce the issue

• GeoTIFF in a private GCS bucket: gs://stac-private-bucket/clouds.tif
• Service account with read access to said bucket: vsigs-issue-service-account.json

Authenticated read works as expected

import json

import osgeo.gdal

with open("vsigs-issue-service-account.json") as fp:
    service_account_json = json.load(fp)

config_opts = {
    "GS_OAUTH2_PRIVATE_KEY": service_account_json["private_key"],
    "GS_OAUTH2_CLIENT_EMAIL": service_account_json["client_email"],    
}

url = "/vsigs/stac-private-bucket/clouds.tif"

with osgeo.gdal.config_options(opts):
    ds = osgeo.gdal.Open(url2)

ds.RasterCount # 1

Authenticated read after an unauthenticated read fails

import json

import osgeo.gdal


# First request will be unsigned
config_opts1 = {
    "GS_NO_SIGN_REQUEST": "YES",
}

with open("vsigs-issue-service-account.json") as fp:
    service_account_json = json.load(fp)

# Second request is signed
config_opts2 = {
    "GS_OAUTH2_PRIVATE_KEY": service_account_json["private_key"],
    "GS_OAUTH2_CLIENT_EMAIL": service_account_json["client_email"],
}

# None of these seem to help either
config_opts3 = {
    **config_opts2,
    "GS_NO_SIGN_REQUEST": "NO",
    "VSI_CACHE": "FALSE",
    "VSI_CACHE_BYTES": "0",
    "CPL_VSIL_CURL_NON_CACHED": "/vsigs/stac-private-bucket/"
}


url = "/vsigs/stac-private-bucket/clouds.tif"

with osgeo.gdal.config_options(config_opts1):
    ds = osgeo.gdal.Open(url) # This fails, as expected

with osgeo.gdal.config_options(config_opts2):
    ds = osgeo.gdal.Open(url) # This also fails

with osgeo.gdal.config_options(config_opts3):
    ds = osgeo.gdal.Open(url) # The extra options also don't help

Versions and provenance

GDAL version: GDAL 3.10.2, released 2025/02/11
Operating system: macOS Sonoma 14.5 (23F79)
Installed via homebrew

Additional context

No response

[arredond]

Update: thanks to this related issue I learnt about osgeo.gdal.VSICurlClearCache(). Calling that function before opening the datasource seems to be an effective workaround 👍