Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Track query and datastore cpu usage metrics #2140

Merged
merged 3 commits into from
Jan 29, 2025

Conversation

joshua-spacetime
Copy link
Collaborator

@joshua-spacetime joshua-spacetime commented Jan 17, 2025

Description of Changes

Tracks cpu usage metrics in both query and datastore. It does not track wasmtime fuel usage

API and ABI breaking changes

None

Expected complexity level and risk

3

The changes are relatively straightforward, just a lot of plumbing.

Testing

An existing query test was updated to test the query metrics. New tests were added for the datastore.

@joshua-spacetime joshua-spacetime requested a review from jsdt January 17, 2025 21:37
@joshua-spacetime joshua-spacetime self-assigned this Jan 17, 2025
@joshua-spacetime joshua-spacetime force-pushed the joshua/materialized-query-execution branch from bc1fcdc to 816f481 Compare January 17, 2025 22:23
@joshua-spacetime joshua-spacetime force-pushed the joshua/query-billing-metrics branch from f493440 to 5bcf8c8 Compare January 17, 2025 22:23
@joshua-spacetime joshua-spacetime force-pushed the joshua/materialized-query-execution branch from 816f481 to 8a36603 Compare January 17, 2025 22:47
@joshua-spacetime joshua-spacetime force-pushed the joshua/query-billing-metrics branch from 5bcf8c8 to 9ede156 Compare January 17, 2025 22:48
@Centril
Copy link
Contributor

Centril commented Jan 17, 2025

How does this affect the subscription benchmarks?

@joshua-spacetime joshua-spacetime force-pushed the joshua/query-billing-metrics branch from 9ede156 to a776af2 Compare January 17, 2025 23:50
@joshua-spacetime
Copy link
Collaborator Author

@Centril the impact for these is negligible.

full-join        time:   [91.830 µs 92.193 µs 92.510 µs]
                 change: [-1.6552% -1.2958% -0.9618%] (p = 0.00 < 0.05)
                 Change within noise threshold.

However once we introduce "bytes scanned" for example, I fully expect the difference will no longer be negligible.

@joshua-spacetime joshua-spacetime force-pushed the joshua/materialized-query-execution branch 2 times, most recently from c33250b to 22af59b Compare January 21, 2025 06:39
@joshua-spacetime joshua-spacetime force-pushed the joshua/query-billing-metrics branch from a776af2 to e7f2b60 Compare January 21, 2025 06:40
Copy link
Contributor

@jsdt jsdt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These metrics look good, though I'm curious about how we will be able to add byte-based read stats.

@joshua-spacetime joshua-spacetime force-pushed the joshua/materialized-query-execution branch from 5ebb2c6 to cc4f397 Compare January 22, 2025 23:09
Base automatically changed from joshua/materialized-query-execution to master January 22, 2025 23:51
@joshua-spacetime joshua-spacetime force-pushed the joshua/query-billing-metrics branch 2 times, most recently from 73988c2 to cc46aad Compare January 23, 2025 05:17
@joshua-spacetime joshua-spacetime changed the title Track cpu usage metrics for query execution Track query and datastore cpu usage metrics Jan 23, 2025
@joshua-spacetime joshua-spacetime force-pushed the joshua/query-billing-metrics branch from 4160fb1 to 0e657af Compare January 23, 2025 23:45
@joshua-spacetime joshua-spacetime requested a review from jsdt January 23, 2025 23:54
Copy link
Contributor

@gefjon gefjon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving as code owner for websocket.rs . Have not checked the rest.

@joshua-spacetime joshua-spacetime force-pushed the joshua/query-billing-metrics branch from 0e657af to e003202 Compare January 28, 2025 17:52
Copy link
Contributor

@jsdt jsdt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding a metric for bytes inserted is the only datastore-related metric that we still need, but that can be in a different PR.

@joshua-spacetime joshua-spacetime force-pushed the joshua/query-billing-metrics branch from e003202 to 7fe9162 Compare January 29, 2025 19:16
@joshua-spacetime joshua-spacetime added this pull request to the merge queue Jan 29, 2025
Merged via the queue into master with commit 4b4484a Jan 29, 2025
13 checks passed
@joshua-spacetime joshua-spacetime deleted the joshua/query-billing-metrics branch January 29, 2025 21:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants