fix(statsd): Emit specific event type tags for "processing.event.produced" metric

[tonyo]

This, for example, fixes the case when transactions are reported as generic "events".

Note: as discussed internally, event_type might not be the best name for this tag anymore, since "event" actually means "error" in some contexts.

[untitaker]

I wonder if there's a way to check whether we have alerts/SLOs set up that filter on this tag

[jan-auer]

as discussed internally, event_type might not be the best name for this tag anymore, since "event" actually means "error" in some contexts.

More so, we use this same tag for every message type that is produced to one of the kafka ingest streams, including metrics, sessions, and user feedback. So far I've been reluctant to change it just to avoid breaking existing alerts.

[jan-auer]

Looks good to me.

Note that this will now create two new tag values besides event: transaction and security. The latter is merely a protocol artifact, but otherwise behaves exactly like event in Kafka and during ingestion.

Depending on what you want to measure here, it seems to me that you actually just want to single out transactions. In that case, you could use the item type to detect transactions, and otherwise keep event.

[mwarkentin]

@tonyo @jan-auer @untitaker I wonder if it would be possible to add the same tag to the processing.produce.error metric?

relay/relay-kafka/src/statsd.rs

Line 19 in 24eea69

Self::ProcessingProduceError => "processing.produce.error",

[jjbayer]

I wonder if it would be possible to add the same tag to the processing.produce.error metric?

@mwarkentin I added a topic tag to that error last week, is that good enough? #3189

[mwarkentin]

@jjbayer I am trying to use these two metrics in combination for an SLO metric in Datadog, so I need the same tag on each - either topic or event_type would work, but right now we have one or the other depending on which metric you're looking at.