Fixing cluster stats response

[l-trotta]

From 0/27 validated responses to 27/27. Providing server side proof for each of these changes would be a lot, most of these changes are adding the in_bytes (or not in_bytes) alternative for fields and making both optional.
New classes:

• GlobalOrdinalsStats -> server code
• ClusterSnapshotStats -> server code
• SearchUsageStats -> server code
• DenseVectorStats -> server code
• SparseVectorStats -> server code

Since it was broken before I think this can all be backported safely

[github-actions]

Following you can find the validation changes for the APIs you have modified.

API	Status	Request	Response
cluster.stats	🔴 → 🟢	27/27	0/27 → 27/27
nodes.stats	🔴	58/58	18/58 → 21/58

You can validate these APIs yourself by using the make validate target.

[l-trotta]

nodes.stats will also be fixed, but this one and the other one need to be merged first

[pquentin]

In SegmentStats, I'm seeing (existing) fields that I would not expect:

• index_writer_max_memory_in_bytes?: long
• stored_memory?: ByteSize

In StatsResponseBase:

• status appears to be optional
• should we add ESQL metrics?

In ClusterFileSystem, there are other optional fields

Still need to finish reviewing types.ts and Stats.ts.

[pquentin]

If I remove this, I get two failures where the health was "unavailable", both in CCS tests (because we don't support CCS with the current test runner). Should we just add unavailable to the list and remove the non_exhaustive annotation?

I'm also not sure about the backwards-compatibility implications.

[l-trotta]

the reason I went for @non_exhaustive is that there are two HealthStatus enums in the server code:

• ClusterHealthStatus: in theory can be only red, green and yellow, but yaml tests show that it can also be "unavailable", mapped with HealthStatus in the spec
• HealthStatus: can be red, green, yellow or unknown, mapped with IndicatorHealthStatus in the spec

In the spec we mostly use HealthStatus while the second one is mapped with the IndicatorHealthStatus enum, which is only used in the health_report endpoint. Since I didn't want to introduce breaking changes by renaming IndicatorHealthStatus and using it to replace HealthStatus when needed, I was thinking of just using @non_exhaustive since I'm not even sure where unavailable comes from, but I agree it's not 100% correct.

Should we introduce the breaking changes for 9.1 to fix every variation of health status?

[l-trotta]

In SegmentStats, I'm seeing (existing) fields that I would not expect:

* index_writer_max_memory_in_bytes?: long

* stored_memory?: ByteSize

nice catch, removing those

In StatsResponseBase:

* `status` appears to be optional

* should we add [ESQL metrics](https://github.com/elastic/elasticsearch/blob/c94bd1cf248bff8e37ea7dc652d19eacfe8fef05/server/src/main/java/org/elasticsearch/action/admin/cluster/stats/ClusterStatsResponse.java#L162-L166)?

status is optional yes, while _esql we already have, it's in ccs

In ClusterFileSystem, there are other optional fields

adding them, thank you!

[github-actions]

Following you can find the validation changes for the APIs you have modified.

API	Status	Request	Response
cluster.stats	🔴 → 🟢	27/27	0/27 → 27/27
nodes.stats	🔴	58/58	18/58 → 21/58

You can validate these APIs yourself by using the make validate target.