Ingest pipeline bug command #153
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hop-dev
added a commit
to elastic/kibana
that referenced
this pull request
Jun 13, 2025
…gine or asset criticality is installed, and add the pipeline to their indices (#221937) Customers who enabled risk scoring in a version below 8.18 and then upgraded to 8.18 or 9.0, will find risk scoring stops working in 0-30 days, this is because the ingest pipeline to set event ingested is not installed, but is specified on the risk scoring index, causing all writes to fail. This PR fixes this by installing the ingest pipeline in all spaces where either: - risk scoring is set up (the risk-engine-configuration saved object is present) - the asset criticality index exists This PR also increases the internal versioning of the asset criticality and risk scoring indices, to trigger the migration which will: - add the pipeline to the asset criticality index - add the pipeline to a risk scoring index which hasnt been rolled over - remove the pipeline from the risk scoring latest index if it is set (if risk scoring was set up in 9.0 or 8.18 for the first time) we remove this because the latest index should not modify the risk scoring documents. The result is that asset criticality and risk scoring should work and event.ingested should correctly be set. ## Other important technical changes - whenevr we increase the risk scoring internal index version, this will now trigger a rollover on the data stream - asset criticality and risk scoring index migrations now support settings changes ## test steps - on 8.17 initialise risk engine and generate some risk scores - upgrade to 8.18.0 or 9.0.0 - rollover the risk score latest datastream dev tool `POST /risk-score.risk-score-default/_rollover` - verify risk scoring has broken, easy way is by assigning asset criticality in the entity flyout - upgrade to 9.0.3 or 8.18.3 - verify risk scoring and asset criticality work ## Test steps for local env: - start kibana - checkout SDG branch elastic/security-documents-generator#153 - run `yarn start risk-score-ingest-bug` AS SUPERUSER - observe asset criticality assignment results in an error / is not performed - restart kibana - observe the ingest pipeline has been created and it works again dev tools verification: ``` # the pipeline should have been created GET /_ingest/pipeline/entity_analytics_create_eventIngest_from_timestamp-pipeline-default # the mapping version should be 4 and the default pipeline should be set GET /.asset-criticality.asset-criticality-default # the mapping version should be 4 and the default pipeline should be set GET /risk-score.risk-score-default # the default pipeline should be set GET /_index_template/.risk-score.risk-score-default-index-template # the default pipeline should NOT be set GET /risk-score.risk-score-latest-default ``` Other verifications: - risk scoring should work - asset criticality assignment should work - ??? --------- Co-authored-by: machadoum <[email protected]> Co-authored-by: Elastic Machine <[email protected]>
hop-dev
added a commit
to hop-dev/kibana
that referenced
this pull request
Jun 13, 2025
…gine or asset criticality is installed, and add the pipeline to their indices (elastic#221937) Customers who enabled risk scoring in a version below 8.18 and then upgraded to 8.18 or 9.0, will find risk scoring stops working in 0-30 days, this is because the ingest pipeline to set event ingested is not installed, but is specified on the risk scoring index, causing all writes to fail. This PR fixes this by installing the ingest pipeline in all spaces where either: - risk scoring is set up (the risk-engine-configuration saved object is present) - the asset criticality index exists This PR also increases the internal versioning of the asset criticality and risk scoring indices, to trigger the migration which will: - add the pipeline to the asset criticality index - add the pipeline to a risk scoring index which hasnt been rolled over - remove the pipeline from the risk scoring latest index if it is set (if risk scoring was set up in 9.0 or 8.18 for the first time) we remove this because the latest index should not modify the risk scoring documents. The result is that asset criticality and risk scoring should work and event.ingested should correctly be set. ## Other important technical changes - whenevr we increase the risk scoring internal index version, this will now trigger a rollover on the data stream - asset criticality and risk scoring index migrations now support settings changes ## test steps - on 8.17 initialise risk engine and generate some risk scores - upgrade to 8.18.0 or 9.0.0 - rollover the risk score latest datastream dev tool `POST /risk-score.risk-score-default/_rollover` - verify risk scoring has broken, easy way is by assigning asset criticality in the entity flyout - upgrade to 9.0.3 or 8.18.3 - verify risk scoring and asset criticality work ## Test steps for local env: - start kibana - checkout SDG branch elastic/security-documents-generator#153 - run `yarn start risk-score-ingest-bug` AS SUPERUSER - observe asset criticality assignment results in an error / is not performed - restart kibana - observe the ingest pipeline has been created and it works again dev tools verification: ``` # the pipeline should have been created GET /_ingest/pipeline/entity_analytics_create_eventIngest_from_timestamp-pipeline-default # the mapping version should be 4 and the default pipeline should be set GET /.asset-criticality.asset-criticality-default # the mapping version should be 4 and the default pipeline should be set GET /risk-score.risk-score-default # the default pipeline should be set GET /_index_template/.risk-score.risk-score-default-index-template # the default pipeline should NOT be set GET /risk-score.risk-score-latest-default ``` Other verifications: - risk scoring should work - asset criticality assignment should work - ??? --------- Co-authored-by: machadoum <[email protected]> Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit d5d37c7) # Conflicts: # x-pack/solutions/security/plugins/security_solution/common/api/quickstart_client.gen.ts # x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/register_entity_analytics_routes.ts # x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/utils/create_ingest_pipeline.ts
hop-dev
added a commit
to hop-dev/kibana
that referenced
this pull request
Jun 13, 2025
…gine or asset criticality is installed, and add the pipeline to their indices (elastic#221937) Customers who enabled risk scoring in a version below 8.18 and then upgraded to 8.18 or 9.0, will find risk scoring stops working in 0-30 days, this is because the ingest pipeline to set event ingested is not installed, but is specified on the risk scoring index, causing all writes to fail. This PR fixes this by installing the ingest pipeline in all spaces where either: - risk scoring is set up (the risk-engine-configuration saved object is present) - the asset criticality index exists This PR also increases the internal versioning of the asset criticality and risk scoring indices, to trigger the migration which will: - add the pipeline to the asset criticality index - add the pipeline to a risk scoring index which hasnt been rolled over - remove the pipeline from the risk scoring latest index if it is set (if risk scoring was set up in 9.0 or 8.18 for the first time) we remove this because the latest index should not modify the risk scoring documents. The result is that asset criticality and risk scoring should work and event.ingested should correctly be set. ## Other important technical changes - whenevr we increase the risk scoring internal index version, this will now trigger a rollover on the data stream - asset criticality and risk scoring index migrations now support settings changes ## test steps - on 8.17 initialise risk engine and generate some risk scores - upgrade to 8.18.0 or 9.0.0 - rollover the risk score latest datastream dev tool `POST /risk-score.risk-score-default/_rollover` - verify risk scoring has broken, easy way is by assigning asset criticality in the entity flyout - upgrade to 9.0.3 or 8.18.3 - verify risk scoring and asset criticality work ## Test steps for local env: - start kibana - checkout SDG branch elastic/security-documents-generator#153 - run `yarn start risk-score-ingest-bug` AS SUPERUSER - observe asset criticality assignment results in an error / is not performed - restart kibana - observe the ingest pipeline has been created and it works again dev tools verification: ``` # the pipeline should have been created GET /_ingest/pipeline/entity_analytics_create_eventIngest_from_timestamp-pipeline-default # the mapping version should be 4 and the default pipeline should be set GET /.asset-criticality.asset-criticality-default # the mapping version should be 4 and the default pipeline should be set GET /risk-score.risk-score-default # the default pipeline should be set GET /_index_template/.risk-score.risk-score-default-index-template # the default pipeline should NOT be set GET /risk-score.risk-score-latest-default ``` Other verifications: - risk scoring should work - asset criticality assignment should work - ??? --------- Co-authored-by: machadoum <[email protected]> Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit d5d37c7) # Conflicts: # x-pack/solutions/security/plugins/security_solution/common/api/quickstart_client.gen.ts # x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/register_entity_analytics_routes.ts # x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/utils/create_ingest_pipeline.ts # x-pack/test/security_solution_api_integration/test_suites/entity_analytics/utils/risk_engine.ts
hop-dev
added a commit
to elastic/kibana
that referenced
this pull request
Jun 13, 2025
…risk engine or asset criticality is installed, and add the pipeline to their indices (#221937) (#223908) # Backport This will backport the following commits from `main` to `8.19`: - [Install event.ingested ingest pipeline in all spaces that the risk engine or asset criticality is installed, and add the pipeline to their indices (#221937)](#221937) <!--- Backport version: 10.0.0 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Mark Hopkin","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-06-13T09:41:40Z","message":"Install event.ingested ingest pipeline in all spaces that the risk engine or asset criticality is installed, and add the pipeline to their indices (#221937)\n\nCustomers who enabled risk scoring in a version below 8.18 and then\nupgraded to 8.18 or 9.0, will find risk scoring stops working in 0-30\ndays, this is because the ingest pipeline to set event ingested is not\ninstalled, but is specified on the risk scoring index, causing all\nwrites to fail.\n\nThis PR fixes this by installing the ingest pipeline in all spaces where\neither:\n- risk scoring is set up (the risk-engine-configuration saved object is\npresent)\n- the asset criticality index exists\n\nThis PR also increases the internal versioning of the asset criticality\nand risk scoring indices, to trigger the migration which will:\n- add the pipeline to the asset criticality index\n- add the pipeline to a risk scoring index which hasnt been rolled over\n- remove the pipeline from the risk scoring latest index if it is set\n(if risk scoring was set up in 9.0 or 8.18 for the first time) we remove\nthis because the latest index should not modify the risk scoring\ndocuments.\n\nThe result is that asset criticality and risk scoring should work and\nevent.ingested should correctly be set.\n\n## Other important technical changes\n\n- whenevr we increase the risk scoring internal index version, this will\nnow trigger a rollover on the data stream\n- asset criticality and risk scoring index migrations now support\nsettings changes\n\n## test steps\n\n- on 8.17 initialise risk engine and generate some risk scores\n- upgrade to 8.18.0 or 9.0.0\n- rollover the risk score latest datastream dev tool `POST\n/risk-score.risk-score-default/_rollover`\n- verify risk scoring has broken, easy way is by assigning asset\ncriticality in the entity flyout\n- upgrade to 9.0.3 or 8.18.3\n- verify risk scoring and asset criticality work\n\n## Test steps for local env:\n\n- start kibana\n- checkout SDG branch\nhttps://github.com/elastic/security-documents-generator/pull/153\n- run `yarn start risk-score-ingest-bug` AS SUPERUSER\n- observe asset criticality assignment results in an error / is not\nperformed\n- restart kibana\n- observe the ingest pipeline has been created and it works again\n\ndev tools verification:\n```\n# the pipeline should have been created\nGET /_ingest/pipeline/entity_analytics_create_eventIngest_from_timestamp-pipeline-default\n\n# the mapping version should be 4 and the default pipeline should be set\nGET /.asset-criticality.asset-criticality-default\n\n# the mapping version should be 4 and the default pipeline should be set\nGET /risk-score.risk-score-default\n\n# the default pipeline should be set\nGET /_index_template/.risk-score.risk-score-default-index-template\n\n# the default pipeline should NOT be set\nGET /risk-score.risk-score-latest-default\n```\n\nOther verifications:\n- risk scoring should work\n- asset criticality assignment should work\n- ???\n\n---------\n\nCo-authored-by: machadoum <[email protected]>\nCo-authored-by: Elastic Machine <[email protected]>","sha":"d5d37c7b2ab9b84d4aafe80b6fdc572b8f95035b","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","backport:prev-minor","Team:Entity Analytics","v9.1.0","backport:8.18","v9.0.3"],"title":"Install event.ingested ingest pipeline in all spaces that the risk engine or asset criticality is installed, and add the pipeline to their indices","number":221937,"url":"https://github.com/elastic/kibana/pull/221937","mergeCommit":{"message":"Install event.ingested ingest pipeline in all spaces that the risk engine or asset criticality is installed, and add the pipeline to their indices (#221937)\n\nCustomers who enabled risk scoring in a version below 8.18 and then\nupgraded to 8.18 or 9.0, will find risk scoring stops working in 0-30\ndays, this is because the ingest pipeline to set event ingested is not\ninstalled, but is specified on the risk scoring index, causing all\nwrites to fail.\n\nThis PR fixes this by installing the ingest pipeline in all spaces where\neither:\n- risk scoring is set up (the risk-engine-configuration saved object is\npresent)\n- the asset criticality index exists\n\nThis PR also increases the internal versioning of the asset criticality\nand risk scoring indices, to trigger the migration which will:\n- add the pipeline to the asset criticality index\n- add the pipeline to a risk scoring index which hasnt been rolled over\n- remove the pipeline from the risk scoring latest index if it is set\n(if risk scoring was set up in 9.0 or 8.18 for the first time) we remove\nthis because the latest index should not modify the risk scoring\ndocuments.\n\nThe result is that asset criticality and risk scoring should work and\nevent.ingested should correctly be set.\n\n## Other important technical changes\n\n- whenevr we increase the risk scoring internal index version, this will\nnow trigger a rollover on the data stream\n- asset criticality and risk scoring index migrations now support\nsettings changes\n\n## test steps\n\n- on 8.17 initialise risk engine and generate some risk scores\n- upgrade to 8.18.0 or 9.0.0\n- rollover the risk score latest datastream dev tool `POST\n/risk-score.risk-score-default/_rollover`\n- verify risk scoring has broken, easy way is by assigning asset\ncriticality in the entity flyout\n- upgrade to 9.0.3 or 8.18.3\n- verify risk scoring and asset criticality work\n\n## Test steps for local env:\n\n- start kibana\n- checkout SDG branch\nhttps://github.com/elastic/security-documents-generator/pull/153\n- run `yarn start risk-score-ingest-bug` AS SUPERUSER\n- observe asset criticality assignment results in an error / is not\nperformed\n- restart kibana\n- observe the ingest pipeline has been created and it works again\n\ndev tools verification:\n```\n# the pipeline should have been created\nGET /_ingest/pipeline/entity_analytics_create_eventIngest_from_timestamp-pipeline-default\n\n# the mapping version should be 4 and the default pipeline should be set\nGET /.asset-criticality.asset-criticality-default\n\n# the mapping version should be 4 and the default pipeline should be set\nGET /risk-score.risk-score-default\n\n# the default pipeline should be set\nGET /_index_template/.risk-score.risk-score-default-index-template\n\n# the default pipeline should NOT be set\nGET /risk-score.risk-score-latest-default\n```\n\nOther verifications:\n- risk scoring should work\n- asset criticality assignment should work\n- ???\n\n---------\n\nCo-authored-by: machadoum <[email protected]>\nCo-authored-by: Elastic Machine <[email protected]>","sha":"d5d37c7b2ab9b84d4aafe80b6fdc572b8f95035b"}},"sourceBranch":"main","suggestedTargetBranches":["9.0"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/221937","number":221937,"mergeCommit":{"message":"Install event.ingested ingest pipeline in all spaces that the risk engine or asset criticality is installed, and add the pipeline to their indices (#221937)\n\nCustomers who enabled risk scoring in a version below 8.18 and then\nupgraded to 8.18 or 9.0, will find risk scoring stops working in 0-30\ndays, this is because the ingest pipeline to set event ingested is not\ninstalled, but is specified on the risk scoring index, causing all\nwrites to fail.\n\nThis PR fixes this by installing the ingest pipeline in all spaces where\neither:\n- risk scoring is set up (the risk-engine-configuration saved object is\npresent)\n- the asset criticality index exists\n\nThis PR also increases the internal versioning of the asset criticality\nand risk scoring indices, to trigger the migration which will:\n- add the pipeline to the asset criticality index\n- add the pipeline to a risk scoring index which hasnt been rolled over\n- remove the pipeline from the risk scoring latest index if it is set\n(if risk scoring was set up in 9.0 or 8.18 for the first time) we remove\nthis because the latest index should not modify the risk scoring\ndocuments.\n\nThe result is that asset criticality and risk scoring should work and\nevent.ingested should correctly be set.\n\n## Other important technical changes\n\n- whenevr we increase the risk scoring internal index version, this will\nnow trigger a rollover on the data stream\n- asset criticality and risk scoring index migrations now support\nsettings changes\n\n## test steps\n\n- on 8.17 initialise risk engine and generate some risk scores\n- upgrade to 8.18.0 or 9.0.0\n- rollover the risk score latest datastream dev tool `POST\n/risk-score.risk-score-default/_rollover`\n- verify risk scoring has broken, easy way is by assigning asset\ncriticality in the entity flyout\n- upgrade to 9.0.3 or 8.18.3\n- verify risk scoring and asset criticality work\n\n## Test steps for local env:\n\n- start kibana\n- checkout SDG branch\nhttps://github.com/elastic/security-documents-generator/pull/153\n- run `yarn start risk-score-ingest-bug` AS SUPERUSER\n- observe asset criticality assignment results in an error / is not\nperformed\n- restart kibana\n- observe the ingest pipeline has been created and it works again\n\ndev tools verification:\n```\n# the pipeline should have been created\nGET /_ingest/pipeline/entity_analytics_create_eventIngest_from_timestamp-pipeline-default\n\n# the mapping version should be 4 and the default pipeline should be set\nGET /.asset-criticality.asset-criticality-default\n\n# the mapping version should be 4 and the default pipeline should be set\nGET /risk-score.risk-score-default\n\n# the default pipeline should be set\nGET /_index_template/.risk-score.risk-score-default-index-template\n\n# the default pipeline should NOT be set\nGET /risk-score.risk-score-latest-default\n```\n\nOther verifications:\n- risk scoring should work\n- asset criticality assignment should work\n- ???\n\n---------\n\nCo-authored-by: machadoum <[email protected]>\nCo-authored-by: Elastic Machine <[email protected]>","sha":"d5d37c7b2ab9b84d4aafe80b6fdc572b8f95035b"}},{"branch":"9.0","label":"v9.0.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> --------- Co-authored-by: kibanamachine <[email protected]>
hop-dev
added a commit
to elastic/kibana
that referenced
this pull request
Jun 13, 2025
…risk engine or asset criticality is installed, and add the pipeline to their indices (#221937) (#223909) # Backport This will backport the following commits from `main` to `8.18`: - [Install event.ingested ingest pipeline in all spaces that the risk engine or asset criticality is installed, and add the pipeline to their indices (#221937)](#221937) <!--- Backport version: 10.0.0 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Mark Hopkin","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-06-13T09:41:40Z","message":"Install event.ingested ingest pipeline in all spaces that the risk engine or asset criticality is installed, and add the pipeline to their indices (#221937)\n\nCustomers who enabled risk scoring in a version below 8.18 and then\nupgraded to 8.18 or 9.0, will find risk scoring stops working in 0-30\ndays, this is because the ingest pipeline to set event ingested is not\ninstalled, but is specified on the risk scoring index, causing all\nwrites to fail.\n\nThis PR fixes this by installing the ingest pipeline in all spaces where\neither:\n- risk scoring is set up (the risk-engine-configuration saved object is\npresent)\n- the asset criticality index exists\n\nThis PR also increases the internal versioning of the asset criticality\nand risk scoring indices, to trigger the migration which will:\n- add the pipeline to the asset criticality index\n- add the pipeline to a risk scoring index which hasnt been rolled over\n- remove the pipeline from the risk scoring latest index if it is set\n(if risk scoring was set up in 9.0 or 8.18 for the first time) we remove\nthis because the latest index should not modify the risk scoring\ndocuments.\n\nThe result is that asset criticality and risk scoring should work and\nevent.ingested should correctly be set.\n\n## Other important technical changes\n\n- whenevr we increase the risk scoring internal index version, this will\nnow trigger a rollover on the data stream\n- asset criticality and risk scoring index migrations now support\nsettings changes\n\n## test steps\n\n- on 8.17 initialise risk engine and generate some risk scores\n- upgrade to 8.18.0 or 9.0.0\n- rollover the risk score latest datastream dev tool `POST\n/risk-score.risk-score-default/_rollover`\n- verify risk scoring has broken, easy way is by assigning asset\ncriticality in the entity flyout\n- upgrade to 9.0.3 or 8.18.3\n- verify risk scoring and asset criticality work\n\n## Test steps for local env:\n\n- start kibana\n- checkout SDG branch\nhttps://github.com/elastic/security-documents-generator/pull/153\n- run `yarn start risk-score-ingest-bug` AS SUPERUSER\n- observe asset criticality assignment results in an error / is not\nperformed\n- restart kibana\n- observe the ingest pipeline has been created and it works again\n\ndev tools verification:\n```\n# the pipeline should have been created\nGET /_ingest/pipeline/entity_analytics_create_eventIngest_from_timestamp-pipeline-default\n\n# the mapping version should be 4 and the default pipeline should be set\nGET /.asset-criticality.asset-criticality-default\n\n# the mapping version should be 4 and the default pipeline should be set\nGET /risk-score.risk-score-default\n\n# the default pipeline should be set\nGET /_index_template/.risk-score.risk-score-default-index-template\n\n# the default pipeline should NOT be set\nGET /risk-score.risk-score-latest-default\n```\n\nOther verifications:\n- risk scoring should work\n- asset criticality assignment should work\n- ???\n\n---------\n\nCo-authored-by: machadoum <[email protected]>\nCo-authored-by: Elastic Machine <[email protected]>","sha":"d5d37c7b2ab9b84d4aafe80b6fdc572b8f95035b","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","backport:prev-minor","Team:Entity Analytics","v9.1.0","backport:8.18","v9.0.3"],"title":"Install event.ingested ingest pipeline in all spaces that the risk engine or asset criticality is installed, and add the pipeline to their indices","number":221937,"url":"https://github.com/elastic/kibana/pull/221937","mergeCommit":{"message":"Install event.ingested ingest pipeline in all spaces that the risk engine or asset criticality is installed, and add the pipeline to their indices (#221937)\n\nCustomers who enabled risk scoring in a version below 8.18 and then\nupgraded to 8.18 or 9.0, will find risk scoring stops working in 0-30\ndays, this is because the ingest pipeline to set event ingested is not\ninstalled, but is specified on the risk scoring index, causing all\nwrites to fail.\n\nThis PR fixes this by installing the ingest pipeline in all spaces where\neither:\n- risk scoring is set up (the risk-engine-configuration saved object is\npresent)\n- the asset criticality index exists\n\nThis PR also increases the internal versioning of the asset criticality\nand risk scoring indices, to trigger the migration which will:\n- add the pipeline to the asset criticality index\n- add the pipeline to a risk scoring index which hasnt been rolled over\n- remove the pipeline from the risk scoring latest index if it is set\n(if risk scoring was set up in 9.0 or 8.18 for the first time) we remove\nthis because the latest index should not modify the risk scoring\ndocuments.\n\nThe result is that asset criticality and risk scoring should work and\nevent.ingested should correctly be set.\n\n## Other important technical changes\n\n- whenevr we increase the risk scoring internal index version, this will\nnow trigger a rollover on the data stream\n- asset criticality and risk scoring index migrations now support\nsettings changes\n\n## test steps\n\n- on 8.17 initialise risk engine and generate some risk scores\n- upgrade to 8.18.0 or 9.0.0\n- rollover the risk score latest datastream dev tool `POST\n/risk-score.risk-score-default/_rollover`\n- verify risk scoring has broken, easy way is by assigning asset\ncriticality in the entity flyout\n- upgrade to 9.0.3 or 8.18.3\n- verify risk scoring and asset criticality work\n\n## Test steps for local env:\n\n- start kibana\n- checkout SDG branch\nhttps://github.com/elastic/security-documents-generator/pull/153\n- run `yarn start risk-score-ingest-bug` AS SUPERUSER\n- observe asset criticality assignment results in an error / is not\nperformed\n- restart kibana\n- observe the ingest pipeline has been created and it works again\n\ndev tools verification:\n```\n# the pipeline should have been created\nGET /_ingest/pipeline/entity_analytics_create_eventIngest_from_timestamp-pipeline-default\n\n# the mapping version should be 4 and the default pipeline should be set\nGET /.asset-criticality.asset-criticality-default\n\n# the mapping version should be 4 and the default pipeline should be set\nGET /risk-score.risk-score-default\n\n# the default pipeline should be set\nGET /_index_template/.risk-score.risk-score-default-index-template\n\n# the default pipeline should NOT be set\nGET /risk-score.risk-score-latest-default\n```\n\nOther verifications:\n- risk scoring should work\n- asset criticality assignment should work\n- ???\n\n---------\n\nCo-authored-by: machadoum <[email protected]>\nCo-authored-by: Elastic Machine <[email protected]>","sha":"d5d37c7b2ab9b84d4aafe80b6fdc572b8f95035b"}},"sourceBranch":"main","suggestedTargetBranches":["9.0"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/221937","number":221937,"mergeCommit":{"message":"Install event.ingested ingest pipeline in all spaces that the risk engine or asset criticality is installed, and add the pipeline to their indices (#221937)\n\nCustomers who enabled risk scoring in a version below 8.18 and then\nupgraded to 8.18 or 9.0, will find risk scoring stops working in 0-30\ndays, this is because the ingest pipeline to set event ingested is not\ninstalled, but is specified on the risk scoring index, causing all\nwrites to fail.\n\nThis PR fixes this by installing the ingest pipeline in all spaces where\neither:\n- risk scoring is set up (the risk-engine-configuration saved object is\npresent)\n- the asset criticality index exists\n\nThis PR also increases the internal versioning of the asset criticality\nand risk scoring indices, to trigger the migration which will:\n- add the pipeline to the asset criticality index\n- add the pipeline to a risk scoring index which hasnt been rolled over\n- remove the pipeline from the risk scoring latest index if it is set\n(if risk scoring was set up in 9.0 or 8.18 for the first time) we remove\nthis because the latest index should not modify the risk scoring\ndocuments.\n\nThe result is that asset criticality and risk scoring should work and\nevent.ingested should correctly be set.\n\n## Other important technical changes\n\n- whenevr we increase the risk scoring internal index version, this will\nnow trigger a rollover on the data stream\n- asset criticality and risk scoring index migrations now support\nsettings changes\n\n## test steps\n\n- on 8.17 initialise risk engine and generate some risk scores\n- upgrade to 8.18.0 or 9.0.0\n- rollover the risk score latest datastream dev tool `POST\n/risk-score.risk-score-default/_rollover`\n- verify risk scoring has broken, easy way is by assigning asset\ncriticality in the entity flyout\n- upgrade to 9.0.3 or 8.18.3\n- verify risk scoring and asset criticality work\n\n## Test steps for local env:\n\n- start kibana\n- checkout SDG branch\nhttps://github.com/elastic/security-documents-generator/pull/153\n- run `yarn start risk-score-ingest-bug` AS SUPERUSER\n- observe asset criticality assignment results in an error / is not\nperformed\n- restart kibana\n- observe the ingest pipeline has been created and it works again\n\ndev tools verification:\n```\n# the pipeline should have been created\nGET /_ingest/pipeline/entity_analytics_create_eventIngest_from_timestamp-pipeline-default\n\n# the mapping version should be 4 and the default pipeline should be set\nGET /.asset-criticality.asset-criticality-default\n\n# the mapping version should be 4 and the default pipeline should be set\nGET /risk-score.risk-score-default\n\n# the default pipeline should be set\nGET /_index_template/.risk-score.risk-score-default-index-template\n\n# the default pipeline should NOT be set\nGET /risk-score.risk-score-latest-default\n```\n\nOther verifications:\n- risk scoring should work\n- asset criticality assignment should work\n- ???\n\n---------\n\nCo-authored-by: machadoum <[email protected]>\nCo-authored-by: Elastic Machine <[email protected]>","sha":"d5d37c7b2ab9b84d4aafe80b6fdc572b8f95035b"}},{"branch":"9.0","label":"v9.0.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> --------- Co-authored-by: kibanamachine <[email protected]>
iblancof
pushed a commit
to iblancof/kibana
that referenced
this pull request
Jun 16, 2025
…gine or asset criticality is installed, and add the pipeline to their indices (elastic#221937) Customers who enabled risk scoring in a version below 8.18 and then upgraded to 8.18 or 9.0, will find risk scoring stops working in 0-30 days, this is because the ingest pipeline to set event ingested is not installed, but is specified on the risk scoring index, causing all writes to fail. This PR fixes this by installing the ingest pipeline in all spaces where either: - risk scoring is set up (the risk-engine-configuration saved object is present) - the asset criticality index exists This PR also increases the internal versioning of the asset criticality and risk scoring indices, to trigger the migration which will: - add the pipeline to the asset criticality index - add the pipeline to a risk scoring index which hasnt been rolled over - remove the pipeline from the risk scoring latest index if it is set (if risk scoring was set up in 9.0 or 8.18 for the first time) we remove this because the latest index should not modify the risk scoring documents. The result is that asset criticality and risk scoring should work and event.ingested should correctly be set. ## Other important technical changes - whenevr we increase the risk scoring internal index version, this will now trigger a rollover on the data stream - asset criticality and risk scoring index migrations now support settings changes ## test steps - on 8.17 initialise risk engine and generate some risk scores - upgrade to 8.18.0 or 9.0.0 - rollover the risk score latest datastream dev tool `POST /risk-score.risk-score-default/_rollover` - verify risk scoring has broken, easy way is by assigning asset criticality in the entity flyout - upgrade to 9.0.3 or 8.18.3 - verify risk scoring and asset criticality work ## Test steps for local env: - start kibana - checkout SDG branch elastic/security-documents-generator#153 - run `yarn start risk-score-ingest-bug` AS SUPERUSER - observe asset criticality assignment results in an error / is not performed - restart kibana - observe the ingest pipeline has been created and it works again dev tools verification: ``` # the pipeline should have been created GET /_ingest/pipeline/entity_analytics_create_eventIngest_from_timestamp-pipeline-default # the mapping version should be 4 and the default pipeline should be set GET /.asset-criticality.asset-criticality-default # the mapping version should be 4 and the default pipeline should be set GET /risk-score.risk-score-default # the default pipeline should be set GET /_index_template/.risk-score.risk-score-default-index-template # the default pipeline should NOT be set GET /risk-score.risk-score-latest-default ``` Other verifications: - risk scoring should work - asset criticality assignment should work - ??? --------- Co-authored-by: machadoum <[email protected]> Co-authored-by: Elastic Machine <[email protected]>
hop-dev
added a commit
to elastic/kibana
that referenced
this pull request
Jun 16, 2025
…isk engine or asset criticality is installed, and add the pipeline to their indices (#221937) (#223905) # Backport This will backport the following commits from `main` to `9.0`: - [Install event.ingested ingest pipeline in all spaces that the risk engine or asset criticality is installed, and add the pipeline to their indices (#221937)](#221937) <!--- Backport version: 10.0.0 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Mark Hopkin","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-06-13T09:41:40Z","message":"Install event.ingested ingest pipeline in all spaces that the risk engine or asset criticality is installed, and add the pipeline to their indices (#221937)\n\nCustomers who enabled risk scoring in a version below 8.18 and then\nupgraded to 8.18 or 9.0, will find risk scoring stops working in 0-30\ndays, this is because the ingest pipeline to set event ingested is not\ninstalled, but is specified on the risk scoring index, causing all\nwrites to fail.\n\nThis PR fixes this by installing the ingest pipeline in all spaces where\neither:\n- risk scoring is set up (the risk-engine-configuration saved object is\npresent)\n- the asset criticality index exists\n\nThis PR also increases the internal versioning of the asset criticality\nand risk scoring indices, to trigger the migration which will:\n- add the pipeline to the asset criticality index\n- add the pipeline to a risk scoring index which hasnt been rolled over\n- remove the pipeline from the risk scoring latest index if it is set\n(if risk scoring was set up in 9.0 or 8.18 for the first time) we remove\nthis because the latest index should not modify the risk scoring\ndocuments.\n\nThe result is that asset criticality and risk scoring should work and\nevent.ingested should correctly be set.\n\n## Other important technical changes\n\n- whenevr we increase the risk scoring internal index version, this will\nnow trigger a rollover on the data stream\n- asset criticality and risk scoring index migrations now support\nsettings changes\n\n## test steps\n\n- on 8.17 initialise risk engine and generate some risk scores\n- upgrade to 8.18.0 or 9.0.0\n- rollover the risk score latest datastream dev tool `POST\n/risk-score.risk-score-default/_rollover`\n- verify risk scoring has broken, easy way is by assigning asset\ncriticality in the entity flyout\n- upgrade to 9.0.3 or 8.18.3\n- verify risk scoring and asset criticality work\n\n## Test steps for local env:\n\n- start kibana\n- checkout SDG branch\nhttps://github.com/elastic/security-documents-generator/pull/153\n- run `yarn start risk-score-ingest-bug` AS SUPERUSER\n- observe asset criticality assignment results in an error / is not\nperformed\n- restart kibana\n- observe the ingest pipeline has been created and it works again\n\ndev tools verification:\n```\n# the pipeline should have been created\nGET /_ingest/pipeline/entity_analytics_create_eventIngest_from_timestamp-pipeline-default\n\n# the mapping version should be 4 and the default pipeline should be set\nGET /.asset-criticality.asset-criticality-default\n\n# the mapping version should be 4 and the default pipeline should be set\nGET /risk-score.risk-score-default\n\n# the default pipeline should be set\nGET /_index_template/.risk-score.risk-score-default-index-template\n\n# the default pipeline should NOT be set\nGET /risk-score.risk-score-latest-default\n```\n\nOther verifications:\n- risk scoring should work\n- asset criticality assignment should work\n- ???\n\n---------\n\nCo-authored-by: machadoum <[email protected]>\nCo-authored-by: Elastic Machine <[email protected]>","sha":"d5d37c7b2ab9b84d4aafe80b6fdc572b8f95035b","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","backport:prev-minor","Team:Entity Analytics","v9.1.0","backport:8.18","v9.0.3"],"title":"Install event.ingested ingest pipeline in all spaces that the risk engine or asset criticality is installed, and add the pipeline to their indices","number":221937,"url":"https://github.com/elastic/kibana/pull/221937","mergeCommit":{"message":"Install event.ingested ingest pipeline in all spaces that the risk engine or asset criticality is installed, and add the pipeline to their indices (#221937)\n\nCustomers who enabled risk scoring in a version below 8.18 and then\nupgraded to 8.18 or 9.0, will find risk scoring stops working in 0-30\ndays, this is because the ingest pipeline to set event ingested is not\ninstalled, but is specified on the risk scoring index, causing all\nwrites to fail.\n\nThis PR fixes this by installing the ingest pipeline in all spaces where\neither:\n- risk scoring is set up (the risk-engine-configuration saved object is\npresent)\n- the asset criticality index exists\n\nThis PR also increases the internal versioning of the asset criticality\nand risk scoring indices, to trigger the migration which will:\n- add the pipeline to the asset criticality index\n- add the pipeline to a risk scoring index which hasnt been rolled over\n- remove the pipeline from the risk scoring latest index if it is set\n(if risk scoring was set up in 9.0 or 8.18 for the first time) we remove\nthis because the latest index should not modify the risk scoring\ndocuments.\n\nThe result is that asset criticality and risk scoring should work and\nevent.ingested should correctly be set.\n\n## Other important technical changes\n\n- whenevr we increase the risk scoring internal index version, this will\nnow trigger a rollover on the data stream\n- asset criticality and risk scoring index migrations now support\nsettings changes\n\n## test steps\n\n- on 8.17 initialise risk engine and generate some risk scores\n- upgrade to 8.18.0 or 9.0.0\n- rollover the risk score latest datastream dev tool `POST\n/risk-score.risk-score-default/_rollover`\n- verify risk scoring has broken, easy way is by assigning asset\ncriticality in the entity flyout\n- upgrade to 9.0.3 or 8.18.3\n- verify risk scoring and asset criticality work\n\n## Test steps for local env:\n\n- start kibana\n- checkout SDG branch\nhttps://github.com/elastic/security-documents-generator/pull/153\n- run `yarn start risk-score-ingest-bug` AS SUPERUSER\n- observe asset criticality assignment results in an error / is not\nperformed\n- restart kibana\n- observe the ingest pipeline has been created and it works again\n\ndev tools verification:\n```\n# the pipeline should have been created\nGET /_ingest/pipeline/entity_analytics_create_eventIngest_from_timestamp-pipeline-default\n\n# the mapping version should be 4 and the default pipeline should be set\nGET /.asset-criticality.asset-criticality-default\n\n# the mapping version should be 4 and the default pipeline should be set\nGET /risk-score.risk-score-default\n\n# the default pipeline should be set\nGET /_index_template/.risk-score.risk-score-default-index-template\n\n# the default pipeline should NOT be set\nGET /risk-score.risk-score-latest-default\n```\n\nOther verifications:\n- risk scoring should work\n- asset criticality assignment should work\n- ???\n\n---------\n\nCo-authored-by: machadoum <[email protected]>\nCo-authored-by: Elastic Machine <[email protected]>","sha":"d5d37c7b2ab9b84d4aafe80b6fdc572b8f95035b"}},"sourceBranch":"main","suggestedTargetBranches":["9.0"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/221937","number":221937,"mergeCommit":{"message":"Install event.ingested ingest pipeline in all spaces that the risk engine or asset criticality is installed, and add the pipeline to their indices (#221937)\n\nCustomers who enabled risk scoring in a version below 8.18 and then\nupgraded to 8.18 or 9.0, will find risk scoring stops working in 0-30\ndays, this is because the ingest pipeline to set event ingested is not\ninstalled, but is specified on the risk scoring index, causing all\nwrites to fail.\n\nThis PR fixes this by installing the ingest pipeline in all spaces where\neither:\n- risk scoring is set up (the risk-engine-configuration saved object is\npresent)\n- the asset criticality index exists\n\nThis PR also increases the internal versioning of the asset criticality\nand risk scoring indices, to trigger the migration which will:\n- add the pipeline to the asset criticality index\n- add the pipeline to a risk scoring index which hasnt been rolled over\n- remove the pipeline from the risk scoring latest index if it is set\n(if risk scoring was set up in 9.0 or 8.18 for the first time) we remove\nthis because the latest index should not modify the risk scoring\ndocuments.\n\nThe result is that asset criticality and risk scoring should work and\nevent.ingested should correctly be set.\n\n## Other important technical changes\n\n- whenevr we increase the risk scoring internal index version, this will\nnow trigger a rollover on the data stream\n- asset criticality and risk scoring index migrations now support\nsettings changes\n\n## test steps\n\n- on 8.17 initialise risk engine and generate some risk scores\n- upgrade to 8.18.0 or 9.0.0\n- rollover the risk score latest datastream dev tool `POST\n/risk-score.risk-score-default/_rollover`\n- verify risk scoring has broken, easy way is by assigning asset\ncriticality in the entity flyout\n- upgrade to 9.0.3 or 8.18.3\n- verify risk scoring and asset criticality work\n\n## Test steps for local env:\n\n- start kibana\n- checkout SDG branch\nhttps://github.com/elastic/security-documents-generator/pull/153\n- run `yarn start risk-score-ingest-bug` AS SUPERUSER\n- observe asset criticality assignment results in an error / is not\nperformed\n- restart kibana\n- observe the ingest pipeline has been created and it works again\n\ndev tools verification:\n```\n# the pipeline should have been created\nGET /_ingest/pipeline/entity_analytics_create_eventIngest_from_timestamp-pipeline-default\n\n# the mapping version should be 4 and the default pipeline should be set\nGET /.asset-criticality.asset-criticality-default\n\n# the mapping version should be 4 and the default pipeline should be set\nGET /risk-score.risk-score-default\n\n# the default pipeline should be set\nGET /_index_template/.risk-score.risk-score-default-index-template\n\n# the default pipeline should NOT be set\nGET /risk-score.risk-score-latest-default\n```\n\nOther verifications:\n- risk scoring should work\n- asset criticality assignment should work\n- ???\n\n---------\n\nCo-authored-by: machadoum <[email protected]>\nCo-authored-by: Elastic Machine <[email protected]>","sha":"d5d37c7b2ab9b84d4aafe80b6fdc572b8f95035b"}},{"branch":"9.0","label":"v9.0.3","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Elastic Machine <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
yarn start risk-score-ingest-bug