Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add auth to V3 monitoring search cursor URLs #10115

Merged
merged 4 commits into from
Aug 9, 2024
Merged

Conversation

joelverhagen
Copy link
Member

@joelverhagen joelverhagen commented Aug 7, 2024

Currently the storage containers for our regional search instances are public. This is okay from a privacy standpoint (all of the content in there is public package metadata). But it is unnecessary. They are really internal storage accounts that should not have external callers. It will make our compliance simpler if we just lock these storage accounts down and use auth in the places that allow access.

Currently, this is our V3 monitoring pipeline, our cursor monitoring system (Synthetics), and NuGet Insights.

Progress on https://github.com/NuGet/Engineering/issues/5576.

This PR adds support for using SAS or managed identity for accessing the Azure Search cursor URLs in the V3 monitoring pipeline.

Summary of changes:

  • Enable base64 decoding on queue client. This is a breaking change in the new SDK (base64 encoding was forced on WindowsAzure.Storage)
  • Add AzureBlobCursor which is a ReadCursor implementation using BlobClient. This allows auth'd scenarios.
  • Fix default storage suffix for creating storage queues. This was broken before.
  • Add -searchCursorSasValue-* and -searchCursorUseManagedIdentity-* arguments to MonitoringProcessor and CatalogToMonitoring so that credentials can be plumbed in for accessing the search cursors.
  • Refactor SearchEndpointConfiguration and related types to allow the various search cursor read scenarios.
  • Improve UT coverage for CommandHelpers.

@joelverhagen joelverhagen requested a review from a team as a code owner August 7, 2024 21:15
@joelverhagen joelverhagen merged commit c0248ba into dev Aug 9, 2024
2 checks passed
@joelverhagen joelverhagen deleted the jver-private-search branch August 9, 2024 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants