[EPIC] Metric Stats

[Dav1dde]

Description

We need outcomes for DDM / custom metrics.

Outcomes for Metrics are generally a tricky problem for several reasons. Metrics are aggregated in multiple stages, SDKs, customer Relays, pop Relays and processing Relays, which makes giving an accurate number of volume hard. On top of that, volume is just a small scaling factor we have to consider, a much bigger factor is the cardinality of a metric.

Ideally outcomes can capture the volume and the cardinality of a metric.

Outcomes should tell us the volume of a single metric (defined by its MRI) and its cardinality per hour.

Our current outcomes cannot capture this information, we need a new mechanism to collect metric outcomes.

Requirements

• Indefinite (?) retention
• Fast enough to query for billing purposes
• Volume and Cardinality needs to be represented
• Sentry UI needs access to show the user
• Billing needs access for billing
• Bizops needs access to process with their own pipelines (can they query snuba instead of reading the topic?)

Why not use Outcomes?

• Outcomes currently don't provide a way to group by metric name (incl. metric type and namespace).
• Cardinality needs to be max() aggregated not sum()'ed

Quantity / Volume

We want to determine the volume of metrics received by the first layer of our infrastructure (PoP-Relays). Client side aggregated metrics are counted with a quantity of 1.

For example: If Relay receives 500 statsd items for a single metric per hour, this metric would be considered to have a volume/quantity of 500 metrics per hour.

Cardinality

We are interested in the cardinality of a single metric (MRI) per hour.

Cardinality can be queried from storage or collected through the Relay cardinality limiter.

Metric Stats Namespaces

Volume: c:metric_stats/volume@none
Tags:

• mri: Metric Name/MRI: <type>:<namespace>/<name>[@<unit>]
• mri.type: Metric Type
• mri.namespace: Metric namespace (extracted from the MRI)
• outcome.id: Outcome ID, metric outcomes share the same numeric outcome ID with regular outcomes.
• outcome.reason: Optional machine readable, free-form reason code.

Cardinality: g:metric_stats/cardinality@none

• mri: Metric Name/MRI: <type>:<namespace>/<name>[@<unit>]
• mri.type: Metric Type
• mri.namespace: Metric namespace (extracted from the MRI)
• cardinality.limit: The cardinality limit id which generated this report.
• cardinality.window: Cardinality window size in seconds.
• cardinality.scope: Cardinality scope (name, project, organization).

If cardinality is tracked by a project or organization the mri* tags will not be present.

Examples

Payloads/Metrics emitted from Relay into the generic metrics topic.

Volume - Accepted

{
   "org_id":0,
   "project_id":42,
   "name":"c:metric_stats/volume@none",
   "type":"c",
   "value":2.0,
   "timestamp":1712223931,
   "tags":{
        "mri": "d:custom/foo@none",
        "mri.type": "d",
        "mri.namespace": "custom",
        "outcome.id": "0",
   },
   "retention_days":90
}

Cardinality by Name

{
   "org_id":0,
   "project_id":42,
   "name":"g:metric_stats/cardinality@none",
   "type":"g",
   "value":{
      "last":2.0,
      "min":2.0,
      "max":2.0,
      "sum":2.0,
      "count":1
   },
   "timestamp":1712223931,
   "tags":{
        "mri": "s:custom/bar@none",
        "mri.type": "s",
        "mri.namespace": "custom",
        "cardinality.limit": "custom-limit-with-some-id",
        "cardinality.scope": "name",
        "cardinality.window": "3600",
   },
   "retention_days":90
}

Cardinality by Type

{
   "org_id":0,
   "project_id":42,
   "name":"g:metric_stats/cardinality@none",
   "type":"g",
   "value":{
      "last":2.0,
      "min":2.0,
      "max":2.0,
      "sum":2.0,
      "count":1
   },
   "timestamp":1712223931,
   "tags":{
        "mri.type": "s",
        "mri.namespace": "custom",
        "cardinality.limit": "custom-limit-with-some-id",
        "cardinality.scope": "type",
        "cardinality.window": "3600",
   },
   "retention_days":90
}

Milestone 1 ✅ - Volume / Happy Path

Implement the happy path ("accepted") for volume metric stats: c:metric_stats/volume@none.

Tasks

Preview Give feedback

1. Create Kafka Topic -> OPS-5396
2. feat(metric-outcomes): Add initial topic and schema sentry-kafka-schemas#231
   +0
   
3. feat(metrics): Add metadata for buckets #3254
   +0
   
4. feat(metric-stats): Emit accepted volume #3281
   +0
   
5. ref(relay): Add metric stats rollout rate to global config sentry#67221
   Scope: Backend +1
   
6. https://github.com/getsentry/sentry-options-automator/pull/929
7. https://github.com/getsentry/ops/pull/9746
8. https://github.com/getsentry/sentry-options-automator/pull/931
9. fix(metric-stats): Prevent producing multiple metric stats for the same bucket #3290
   +0
   
10. ref(metrics): Add metric_stats usecase id to entity mapping sentry#67596
    Scope: Backend +1
    

Milestone 2 ✅ - Cardinality / Happy Path

Implement the happy path ("accepted") for cardinality metric stats: g:metric_stats/cardinality@none.

Tasks

Preview Give feedback

1. feat(cardinality): Implement name based cardinality limits #3313
   +0
   
2. ref(cardinality): Pipeline Redis script invocations #3321
   +0
   
3. feat(cardinality): Implement cardinality reporting #3342
   +0
   
4. feat(metric-stats): Report cardinality to metric stats #3360
   +0
   
5. ref(relay): Make cardinality limits more configurable sentry#68158
   Scope: Backend +1
   
6. https://github.com/getsentry/sentry-options-automator/pull/1033

Milestone 2.5 ✅ - Cardinality by Minute and Project

Tasks

Preview Give feedback

1. feat(metric-stats): Add additional tags to metric stats metrics #3438
   +0
   
2. feat(metric-stats): Add cardinality.{limit,scope} tags to strings indexer sentry#68891
   Scope: Backend +1
   
3. feat(metric-stats): Implement per type cardinality limits #3437
   +0
   
4. Enable per minute cardinality

Milestone 3 ✅ - Negative Outcomes

Tasks

Preview Give feedback

1. feat(ddm): Add negative outcomes to metric stats in pops #3452
   +0
2. feat(metric-stats): Add cardinality limited outcome id #3389
   +0
   
3. feat(metric-stats): Add cardinality limited outcome id sentry#68422
   Scope: Backend Scope: Frontend +2
   
4. Emit Negative Outcomes: Cardinality Limits #3433
   +0
   
5. Emit Negative Outcomes: Rate Limits #3434
   +0
   
6. Emit Negative Outcomes: Throughput Limiter #3435
   +0
   
7. Emit Negative Outcomes: Denied Metrics #3482
   +0
   
8. feat(metric-stats): Emit negative outcomes of metrics in metric stats #3436
   +0
   

Milestone 4 - Finishing Touches

Tasks

Preview Give feedback

1. ref(metrics): Split project metric transforms into separate steps #3565
   +0
   
2. ref(metrics): Make buckets view generic over the contained buckets #3569
   +0
   
3. https://github.com/getsentry/ops/pull/10501
4. https://github.com/getsentry/ops/pull/10345
5. fix(metric-stats): Use cardinality timestamp for reports #3626
   +0
   
6. ref(metric-stats): Make metric stats collection configurable #3629
   +0
   
7. Configurable/Longer Retention
8. Check for missing integration tests (prime candidate: rate limits -> negative outcomes)
9. Come up with a better way to reject metrics than our current util function
10. Figure out BucketViews and volume
11. check_buckets drops buckets without reject_metrics
12. Cardinality Limits are overlapping, if a limit rejects a metric, all other limits are still increased
13. fix(metric-stats): Only emit volume metadata for the first bucket view #3609
    +0
    
14. fix(metric-stats): Fix metric stats outcomes for global metrics endpoint #3612
    +0
    
15. ref(metric-stats): Rate limit before cardinality limiting #3582
    +0
    

[mcannizz]

@Dav1dde I am wondering if this work is still in flight, and whether we consider that work to be P0. It seems like the remaining tasks are lower priority. If you agree, could you please move the remaining stuff to a lower priority epic and close this one?

[Dav1dde]

I would like to keep the Epic open, we are at a stage where we provide all the data which is needed now but we're not at a stage that this data will stay correct with upcoming features.
E.g. currently metric_stats does not interact well with user defined cardinality limits which the product is already planning.

Basically at this point we implemented the minimum set everyone else needs but the remaining legwork in Relay is as important just not visible to anyone outside of ingest.

That being said, if moving the remaining tasks to a new Epic makes stuff easier for you I am not opposed to it.

[mcannizz]

@Dav1dde makes sense, thanks. I'm going to leave this epic as is but reduce it's priority since it sounds like we've completed the P0 work. Feel free to adjust as you see fit.

[Dav1dde]

Closing this, we got all the important bits in place and with the current restructure of the metrics product the remaining pieces are not as important anymore.