[docs] Migrate docs from AsciiDoc to Markdown

[colleenmcginnis]

Migrate docs from AsciiDoc to Markdown.
Please see docs preview

Notes:

• I deleted AsciiDoc and image files in each {beat}/docs/ directory.
• I'm not sure what to do with the AsciiDoc changelog files.

For reference, here are details about the docs migration.

[kilfoyle]

Reviewers, please let me know if you need any details about this. I assume that the failing CI checks don't matter, or the tests can be removed since they won't be applicable any longer with the new Markdown docs.

Also, just for awareness, we'll have a 1:1 mapping so that each HTML docs page now has a single Markdown source file. Previously, changes in the 'libbeat' directory were propagated to the Metricbeat docs, Filebeat docs, etc. We're making similar changes across all of the docs to reduce complexity.

[belimawr]

@belimawr Do you know if we still have generated docs? If that's still needed, I can open an issue to ask the Docs Engineering team about converting the generation scripts to output Markdown instead of Asciidoc.

@kilfoyle, yes, we still have generated docs and as far as I can tell we still need them.

While the Generated docs is a bit outdated (e.g: docs.asciidoc is not used/does not exist any more), the generated files listed there are still generated by the listed scripts.

So we have to update them to output Markdown and to stop relying on reading versions from libbeat/docs/version.asciidoc.

[kilfoyle]

Thanks @belimawr! If I should open an issue for those changes or if there's anything else I can do, please let me know.

[belimawr]

Thanks @belimawr! If I should open an issue for those changes or if there's anything else I can do, please let me know.

Yes, please, could you open an issue for that?

Given your comment:

If that's still needed, I can open an issue to ask the Docs Engineering team about converting the generation scripts to output Markdown instead of Asciidoc.

I understood you'd open the issue.

Honestly, I'm not sure who is responsible for those scripts: the Data-Plane (Beats) team or the Docs team 🙈.

Anyway, if there is anything I can help with, just ping me ;)

[bmorelli25]

Do you know if we still have generated docs? If that's still needed, I can open an issue to ask the Docs Engineering team about converting the generation scripts to output Markdown instead of Asciidoc.

Unfortunately, the Docs Eng team doesn’t have the bandwidth to update automation scripts for every engineering team 😞 . We know this work takes time to prioritize and schedule, so we can ship static docs (this point-in-time snapshot) for teams that need more runway while they update their automation. Ideally, we'd be able to do that from this repo. But it sounds like there’s some automation that needs fixing before we can merge this PR.

@kilfoyle is meeting with @pierrehilbert on Friday. I've asked to be added to that meeting to continue the discussion.

[kilfoyle]

Thanks a lot @belimawr for the investigation and @bmorelli25 for the clarification! And sorry for my confusion: I wasn't sure which team should or could own this work.

I've opened #43008 to see if we can:

1. Remove or fix the dependencies on libbeat/docs/version.asciidoc that cause the CI checks to fail.
2. Update the script that produces the generated docs to output Markdown instead of Asciidoc.

If I've missed any info in the issue please feel free to make any additions/corrections.

On Friday, Brandon and I will chat with Pierre about this issue and also the Beats Release Notes.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b migrate-docs upstream/migrate-docs
git merge upstream/main
git push upstream migrate-docs

[kilfoyle]

Just to try and get some of the CI checks to pass I've temporarily re-added:

• libbeat/docs/release.asciidoc
• libbeat/docs/version.asciidoc

Unfortunately I don't think it helped.

[colleenmcginnis]

👋 @kilfoyle this is just a guess, but it looks like there were quite a few non-AsciiDoc files deleted (that's on me). I can try restoring them to see if that fixes the checks.

[bmorelli25]

Thanks, @colleenmcginnis! I'm seeing helpful error messages now about missing files used for docs automation. As a temporary solution to help us get this PR merged, I'm going to work on restoring the .asciidoc files that are used for automation purposes. That should help these checks pass.

[bmorelli25]

I could use some help from a Beats engineer on understanding why golangci-lint / lint is failing even though none of those files are changed by this PR.

[belimawr]

I could use some help from a Beats engineer on understanding why golangci-lint / lint is failing even though none of those files are changed by this PR.

This check is not required. It checks every file touched by the PR and will give warnings even if the lines were not added by the PR.

If it is waning about lines you didn't touch, you can just ignore it.