Skip to content

ingest pipeline docs "ingest lag" #1672

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 26 commits into
base: main
Choose a base branch
from

Conversation

philippkahr
Copy link
Contributor

@philippkahr philippkahr commented Jun 10, 2025

#1381 relates to this, we pulled this file out as it's standalone for better review

Copy link

github-actions bot commented Jun 12, 2025

🔍 Preview links for changed docs:

🔔 The preview site may take up to 3 minutes to finish building. These links will become live once it completes.


The basic calculation is:

`event.ingested - @timestamp`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since @timestamp is usually overrode to data source timestamp, my team usually recommends event.created instead which is when it hits its first Elastic product (although I believe exception is Logstash pipelines don't qualify as pipeline in that doc and don't induce the field although Agent+Elasticsearch would.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but event.created is not created automatically in Elastic Agent anymore, thus you need to add this to all the integrations manually. We can add a recommendation for it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I rewrote it a bit and added the code needed to add the event.created.

Copy link
Contributor

@theletterf theletterf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First review pass.

@@ -95,6 +95,7 @@ toc:
- file: ingest/upload-data-files.md
- file: ingest/transform-enrich.md
children:
- file: ingest/transform-enrich/ingest-lag.md
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should edit https://www.elastic.co/docs/manage-data/ingest/transform-enrich to add the appropriate section for the new doc.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No idea what do you mean by that to be honest. Can you take care of that for me?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You'll see that in https://www.elastic.co/docs/manage-data/ingest/transform-enrich each of the subpages is introduced. Not mandatory, but might worth doing it for this one as well. A paragraph could suffice.

philippkahr and others added 17 commits June 23, 2025 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants