fix(spans): Align the spans topics with reality

[jan-auer]

Various fixes to the ingest-spans, buffered-segments, and snuba-spans
schema definitions:

• Remove _metrics_summary, as this is no longer produced and supported
• Make parent_span_id and segment_id optional everywhere
• Remove exclusive_time_ms from ingest-spans and buffered-segments
• Remove is_segment and segment_id from ingest-spans

Also exposes SegmentSpan as named type from buffered-segments for the consumer

[github-actions]

versions in use:

The following repositories use one of the schemas you are editing. It is recommended to roll out schema changes in small PRs, meaning that if those used versions lag behind the latest, it is probably best to update those services before rolling out your change.

latest version: 1.1.1

benign changes

schemas/buffered-segments.v1.schema.json

• Removed a property _metrics_summary from .spans.?, but it is still accepted via additionalProperties=true

  {"path": ".spans.?", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "_metrics_summary"}}}
  

• Removed a property exclusive_time_ms from .spans.?, but it is still accepted via additionalProperties=true

  {"path": ".spans.?", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "exclusive_time_ms"}}}
  

• {"path": ".spans.?.parent_span_id", "change": {"TypeAdd": {"added": "null"}}}
  

• {"path": ".spans.?.segment_id", "change": {"TypeAdd": {"added": "null"}}}
  

• {"path": ".spans.?", "change": {"RequiredRemove": {"property": "exclusive_time_ms"}}}
  

schemas/ingest-spans.v1.schema.json

• Removed a property _metrics_summary from ``, but it is still accepted via additionalProperties=true

  {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "_metrics_summary"}}}
  

• Removed a property exclusive_time_ms from ``, but it is still accepted via additionalProperties=true

  {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "exclusive_time_ms"}}}
  

• Removed a property is_segment from ``, but it is still accepted via additionalProperties=true

  {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "is_segment"}}}
  

• Removed a property segment_id from ``, but it is still accepted via additionalProperties=true

  {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "segment_id"}}}
  

• {"path": ".parent_span_id", "change": {"TypeAdd": {"added": "null"}}}
  

• {"path": "", "change": {"RequiredRemove": {"property": "exclusive_time_ms"}}}
  

• {"path": "", "change": {"RequiredRemove": {"property": "is_segment"}}}
  

schemas/snuba-spans.v1.schema.json

• Removed a property _metrics_summary from ``, but it is still accepted via additionalProperties=true

  {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "_metrics_summary"}}}
  

• {"path": ".parent_span_id", "change": {"TypeAdd": {"added": "null"}}}
  

• {"path": ".segment_id", "change": {"TypeAdd": {"added": "null"}}}
  

✅ This PR should be safe to roll out to consumers first. Make sure to bump
the library in the following repos first:

getsentry/sentry
getsentry/snuba
getsentry/super-big-consumers

...then in the other repos:

getsentry/sentry
getsentry/relay

Take a look at the README for how to release a new version of sentry-kafka-schemas.

[kylemumma]

• What are these files? Are these schema definitions for what data looks like on a kafka topic?
• What is the difference between the 3: buffered_segment, ingest-spans, and snuba-spans? The only topic I am familiar with is the snuba-spans topic that snuba consumes spans off of.

[jan-auer]

• What are these files? Are these schema definitions for what data looks like on a kafka topic?

These files define minimum requirements for valid data on the topics required by the consuming side. The schema can be enforced in consumers. I'm not sure if Snuba uses them, at least Sentry does. @untitaker may have more information on that.

• What is the difference between the 3: buffered_segment, ingest-spans, and snuba-spans? The only topic I am familiar with is the snuba-spans topic that snuba consumes spans off of.

snuba-spans is the only topic we use in production today. Relay produces and Snuba consumes. We're preparing a change in the pipeline that introduces two more intermediate steps, hence the new topics:

• Relay produces to ingest-spans and the trace buffer in Sentry consumes.
• The trace buffer produces to buffered-segments and a segment processor consumes.
• The segment consumer produces to snuba-spans, which Snuba/EAP consume from.

There are differences between these topics, as each of the pipeline steps adds more attributes to the spans. From Snuba's perspective nothing changes, as Snuba will still consume the same payloads from snuba-spans.

The reason for the change in this schema is that it was incorrect. Specifically, parent_span_id and segment_id can be null. My understanding is that the Snuba consumer already handles this case, as Relay can produce such values, but I'd ask for confirmation on that.

[kylemumma]

The change to snuba-spans to allow parent_span_id and segment_id to be null should be safe for snuba consumers because they already handle the null case for these fields. https://github.com/getsentry/snuba/blob/b6f874b16bc533ae233ebfac39110c3d473bc907/rust_snuba/src/processors/spans.rs#L39-L62

[kylemumma]

why is this removed?

[jan-auer]

See the PR description - metrics summaries do not exist anymore and will not come back.

[kylemumma]

why are these removed?