Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(provider-index): parallelize claim lookups #121

Merged
merged 3 commits into from
Feb 12, 2025

Conversation

frrist
Copy link
Member

@frrist frrist commented Feb 12, 2025

via 656fa67

  • ProviderIndexService.getProviderResults now queries IPNI and legacy claims concurrently using channels. If IPNI returns a valid result, the legacy lookup is canceled.
  • errors from both queries are joined to provide better context when neither returns data.
  • tests have been updated to cover these new scenarios.
  • closes Parallelize ProviderIndexService Find Method #120

via ae21e44

  • adds a few more events to the span for the operation be modified allowing timing of various operations to be inferred. If we prefer this change be landed separately (or not at all!) happy to revert it.

@frrist frrist self-assigned this Feb 12, 2025
Copy link

codecov bot commented Feb 12, 2025

Codecov Report

Attention: Patch coverage is 96.77419% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
pkg/service/providerindex/providerindex.go 96.77% 1 Missing and 1 partial ⚠️
Files with missing lines Coverage Δ
pkg/service/providerindex/providerindex.go 62.85% <96.77%> (+7.24%) ⬆️

... and 1 file with indirect coverage changes

@frrist frrist marked this pull request as ready for review February 12, 2025 00:30
@frrist frrist requested a review from hannahhoward February 12, 2025 00:31
frrist added 2 commits February 11, 2025 16:33
- ProviderIndexService.getProviderResults now queries IPNI and legacy claims concurrently using channels.
  If IPNI returns a valid result, the legacy lookup is canceled, reducing unnecessary wait times.
- errors from both queries are joined to provide better context when neither returns data.
- tests have been updated to cover these new scenarios.
- changes permit:
  - runtime of cache lookup to be inferred.
  - runtimes of ipni & legacy fetch operations to be inferred.
@frrist frrist force-pushed the frrist/feat/providerindex/parallelquery branch from 494404f to ae21e44 Compare February 12, 2025 00:34
Copy link
Member

@hannahhoward hannahhoward left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@volmedo volmedo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. It may be valuable to add a couple more tests for the cases where IPNI and/or legacy return no results and no errors.

@frrist
Copy link
Member Author

frrist commented Feb 12, 2025

@volmedo great idea 😄

  • I've added a couple new tests via 13931b0 that cover the case when IPNI/Legacy return a result and Legacy/IPNI return an empty result (but no error).
  • Coverage for neither returning a result nor error is (thankfully) already well covered here:
    t.Run("returns an empty slice when results are not found anywhere, nothing gets cached", func(t *testing.T) {
    mockStore := types.NewMockProviderStore(t)
    mockIpniFinder := extmocks.NewMockIpniFinder(t)
    mockIpniPublisher := extmocks.NewMockIpniPublisher(t)
    mockLegacyClaims := NewMockLegacyClaimsFinder(t)
    providerIndex := New(mockStore, mockIpniFinder, mockIpniPublisher, mockLegacyClaims)
    someHash := testutil.RandomMultihash()
    mockStore.EXPECT().Members(testutil.AnyContext, someHash).Return(nil, types.ErrKeyNotFound)
    mockIpniFinder.EXPECT().Find(testutil.AnyContext, someHash).Return(&model.FindResponse{}, nil)
    mockLegacyClaims.EXPECT().Find(testutil.AnyContext, someHash, []multicodec.Code{0}).Return([]model.ProviderResult{}, nil)
    results, err := providerIndex.getProviderResults(context.Background(), someHash, []multicodec.Code{0})
    require.NoError(t, err)
    require.Empty(t, results)
    })

@frrist frrist merged commit feb3f7b into main Feb 12, 2025
9 checks passed
@frrist frrist deleted the frrist/feat/providerindex/parallelquery branch February 12, 2025 20:47
@volmedo
Copy link
Member

volmedo commented Feb 13, 2025

thanks for adding the extra tests @frrist! 💯

A nitpick that we may want to implement at some point: I spotted a couple comments that I think should read different (in lines 321 and 371). They mention that an error is returned when it's in fact an empty result.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Parallelize ProviderIndexService Find Method
3 participants