[Pull-based Ingestion] Add support for dynamically updating ingestion error handling strategy

[varunbharadwaj]

Description

1. This PR is a follow up for [Pull-based Ingestion] Add error handling strategy to pull-based ingestion #17427 to add support for dynamically updating ingestion error strategy using update_settings API.
2. Message processor will indefinitely retry failed messages after a wait time, if a BLOCK error strategy is used. Updating to DROP strategy will skip the failed messages.
3. Additionally, the PR fixes initial global checkpoint in p2p segRep mode which is validated by flows such as CloseIndex API.

This PR forms the base on which subsequent PRs will build on for adding pause/resume APIs.

Related Issues

Resolves part of #17442. Subsequent PRs will add pause/resume APIs.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 92b576e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 9f94093: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 71.79487% with 11 lines in your changes missing coverage. Please review.

Project coverage is 72.41%. Comparing base (2ee8660) to head (589c235).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...ndices/pollingingest/MessageProcessorRunnable.java	72.22%	3 Missing and 2 partials ⚠️
...a/org/opensearch/index/engine/IngestionEngine.java	55.55%	4 Missing ⚠️
...in/java/org/opensearch/index/shard/IndexShard.java	50.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #17565      +/-   ##
============================================
- Coverage     72.43%   72.41%   -0.02%     
- Complexity    65694    65719      +25     
============================================
  Files          5311     5311              
  Lines        304937   304962      +25     
  Branches      44226    44227       +1     
============================================
- Hits         220872   220842      -30     
- Misses        65912    66058     +146     
+ Partials      18153    18062      -91     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[andrross]

Pause message processor when an error is encountered on BLOCK error strategy.

Would it be better to keep retrying indefinitely at some fixed rate? It can be hard to determine if errors are transient or not, and in this case if you enter the paused state because of a transient failure you'll just be stuck until something intervenes, right?

[varunbharadwaj]

Pause message processor when an error is encountered on BLOCK error strategy.

Would it be better to keep retrying indefinitely at some fixed rate? It can be hard to determine if errors are transient or not, and in this case if you enter the paused state because of a transient failure you'll just be stuck until something intervenes, right?

Good point, as discussed today, updated to indefinitely retry and skip only after user switches to a DROP policy.

[github-actions]

❌ Gradle check result for e6e34ed: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 5c0788c: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 589c235: SUCCESS