Ingest pipeline best practices #1381
Conversation
|
There are a couple of things I need help with.
|
philippkahr
added
Team:Platform
kilfoyle
added
Team:Obs
|
Thanks a lot for opening this Philipp! I've added the "Team:Obs" label since under the new docs organization that's where the ingest content will land. |
There was a problem hiding this comment.
@philippkahr I started reviewing this PR, but I didn't get very far (yet!). There's a lot of content to get into! I'm just going to post the comments/questions/suggestions I have so far so I can see if I'm on the right track. I can jump back in next week.
Some themes in my early feedback include:
- I see some opportunities to simplify the examples to really emphasize the point you're making in each section.
- It might be helpful to write out in plain language what the example is trying to achieve before jumping into a code snippet. (I provided a couple suggestions below.)
- There are probably opportunities to remove redundant information.
| ### Contains operation and null check | ||
|
|
||
| This includes an initial null check, which is not necessary. | ||
|
|
||
| ```painless | ||
| "if": "ctx.event?.action !=null | ||
| && ['bandwidth','spoofed syn flood prevention','dns authentication','tls attack prevention', | ||
| 'tcp syn flood detection','tcp connection limiting','http rate limiting', | ||
| 'block malformed dns traffic','tcp connection reset','udp flood detection', | ||
| 'dns rate limiting','malformed http filtering','icmp flood detection', | ||
| 'dns nxdomain rate limiting','invalid packets'].contains(ctx.event.action)" | ||
| ``` | ||
|
|
||
| This behaves nearly the same: | ||
|
|
||
| ```painless | ||
| "if": "['bandwidth','spoofed syn flood prevention','dns authentication','tls attack prevention', | ||
| 'tcp syn flood detection','tcp connection limiting','http rate limiting', | ||
| 'block malformed dns traffic','tcp connection reset','udp flood detection', | ||
| 'dns rate limiting','malformed http filtering','icmp flood detection', | ||
| 'dns nxdomain rate limiting','invalid packets'].contains(ctx.event?.action)" | ||
| ``` | ||
|
|
||
| The difference is in the execution itself which should not matter since it is Java under the hood and pretty fast as this. In reality what happens is the following when doing the first one with the initial: `ctx.event?.action != null` If action is null, then it will exit here and not even perform the contains operation. In our second example we basically run the contains operation x times, for every item in the array and have `valueOfarray.contains('null')` then. |
There was a problem hiding this comment.
This example confuses me. Why would you want to run the contains operation n times if you already know ctx.event.action is null and it's going to return false.
|
☝️ Updating with the latest on |
|
Cross linking with #1727, so we can ensure the new docs complement each other properly. |
| } | ||
| } | ||
| ], | ||
| "on_failure": [ |
There was a problem hiding this comment.
Feel free to ignore, but IME when users want global error handling they also want to reroute documents to a different failure index which might be nice to include.
|
|
||
| There are various ways to handle data in ingest pipelines, and while they all produce similar results, some methods might be more suitable depending on the specific case. This section provides guidance to ensure that your ingest pipelines are consistent, readable, and maintainable. While we won't focus heavily on performance optimizations, the goal is to create pipelines that are easy to understand and manage. | ||
|
|
||
| ## Accessing Fields in `if` Statements |
There was a problem hiding this comment.
"if statements" are officially called conditionals
AND I'd like to request this be flushed out to include context:
- that errors inside
ifs don't go toignore_failureConditional ingest processors should respectignore_failurefor errors inifcondition elasticsearch#126005 - cross-linking to top support volume is Painless exception handling for missing keys & missing values
There was a problem hiding this comment.
@colleenmcginnis can you help me cross link this page https://github.com/elastic/docs-content/pull/1381/files#diff-a70e1482d1c5914e741755809833f482be84ef173010439dae33fb194fd57a55 the error-handling one then here ? I added a bit of fluff around it.
|
|
||
| ```painless | ||
| if (ctx.user_name != null) { | ||
| ctx.user.name = ctx.user_name |
There was a problem hiding this comment.
per above this if will error if field doesn't exist (vs check is field exists but its value is NULL)
There was a problem hiding this comment.
I don#t fully get it. Because this:
POST _ingest/pipeline/_simulate
{
"docs": [
{
"_source": {
"user_name": ""
}
},
{
"_source": {
"user_name": "abc"
}
},
{
"_source": {
}
}],
"pipeline": {
"processors": [
{
"script": {
"source": "
ctx.user = new HashMap();
ctx.user.name = ctx.user_name;
"
}
}
]
}
}
works as expected, the value of user.name is then actually: null. Or am I missing something?
|
🔍 Preview links for changed docs:
🔔 The preview site may take up to 3 minutes to finish building. These links will become live once it completes. |
…-pipelines Suggested copy edits for `manage-data/ingest/transform-enrich/common-mistakes.md`
|
|
||
| Ingest pipelines in Elasticsearch are powerful tools for transforming and enriching data before indexing. However, errors can occur during processing. This guide outlines strategies for handling such errors effectively. | ||
|
|
||
| **Important**: Ingest pipelines are executed before the document is indexed by Elasticsearch. You can handle the errors occurring while processing the document (i.e. transforming the json object) but not the errors triggered while indexing like mapping conflict. For this is the Elasticsearch Failure Store. |
There was a problem hiding this comment.
| **Important**: Ingest pipelines are executed before the document is indexed by Elasticsearch. You can handle the errors occurring while processing the document (i.e. transforming the json object) but not the errors triggered while indexing like mapping conflict. For this is the Elasticsearch Failure Store. | |
| :::{important} | |
| Ingest pipelines are executed before the document is indexed by Elasticsearch. You can handle the errors occurring while processing the document (i.e. transforming the json object) but not the errors triggered while indexing like mapping conflict. For this is the Elasticsearch Failure Store. | |
| ::: |
| - Parsing Errors: Occur when a processor fails to parse a field, such as a date or number. | ||
| - Missing Fields: Happen when a required field is absent in the document. | ||
|
|
||
| :::tip |
There was a problem hiding this comment.
This should fix the rendering issue (seen here).
| :::tip | |
| :::{tip} |
|
|
||
| The `on_failure` parameter can be defined either for individual processors or at the pipeline level to catch exceptions that may occur during document processing. The `ignore_failure` option allows a specific processor to silently skip errors without affecting the rest of the pipeline. | ||
|
|
||
| ## Global vs. Processor-Specific |
There was a problem hiding this comment.
| ## Global vs. Processor-Specific | |
| ## Global vs. processor-specific |
|
|
||
| We can restructure the pipeline by moving the `on_failure` handling directly into the processor itself. This allows the pipeline to continue execution. In this case, the `event.category` processor still runs. You can also retain the global `on_failure` to handle errors from other processors, while adding processor-specific error handling where needed. | ||
|
|
||
| (While executing two `set` processors within the `dissect` error handler may not always be ideal, it serves as a demonstration.) |
There was a problem hiding this comment.
Maybe use a note here instead of a paragraph in parentheses?
There was a problem hiding this comment.
This file needs to be significantly edited down. There's a lot of overlap with the content in the Create readable and maintainable ingest pipelines page (manage-data/ingest/transform-enrich/common-mistakes.md). We should avoid duplication to avoid confusion among users and make the docs easier to maintain. Let me know if you'd like me to take a pass at de-duplicating this content.
| serverless: ga | ||
| --- | ||
|
|
||
| # Tips and Tricks |
There was a problem hiding this comment.
I'm not really a fan of this as a title for technical documentation, but I can't think of a good alternative at the moment. 🙃
|
|
||
| There are various ways to handle data in ingest pipelines, and while they all produce similar results, some methods might be more suitable depending on the specific case. This section provides guidance to ensure that your ingest pipelines are consistent, readable, and maintainable. While we won't focus heavily on performance optimizations, the goal is to create pipelines that are easy to understand and manage. | ||
|
|
||
| ## Accessing fields in `if` statements / conditionals |
There was a problem hiding this comment.
This section seems to duplicate information from the Create readable and maintainable ingest pipelines page. Do we need this information in two places?
|
|
||
| This works fine, as you now check for null. | ||
|
|
||
| However there is also an easier to write and maintain alternative available: |
There was a problem hiding this comment.
This also feels like it's aligned with the content in Create readable and maintainable ingest pipelines. Could we integrate this information into that file using the format established there?
|
|
||
| ## Remove empty fields or remove empty fields that match a regular expression | ||
|
|
||
| Alex and Honza created a [blog post](https://alexmarquardt.com/2020/11/06/using-elasticsearch-painless-scripting-to-iterate-through-fields/) presenting painless scripts that remove empty fields or fields that match a regular expression. We are already using this in a lot of places. Most of the time in the custom pipeline and in the final pipeline as well. |
There was a problem hiding this comment.
I don't think we should rely on a link to an external blog post in the official documentation. We don't have control over whether that information will continue to be available and will be kept up to date.
| } | ||
| ``` | ||
|
|
||
| ## Check if a value exists and is not null |
There was a problem hiding this comment.
Again, this feels like it's duplicating a lot of the same information in the Create readable and maintainable ingest pipelines page.
based on the discussions here: #1052
this is my first PR against the docs, and I am building a couple of new pages. I think it makes sense to split it out. I am putting it into that part of the docs. https://www.elastic.co/docs/manage-data/ingest/transform-enrich/ingest-pipelines The tips and tricks are generic and not specific to just o11y, or security.