Add ability to call generate commands on asset units #1185

panglesd · 2024-08-01T13:31:48Z

The pipeline for asset was missing the generate command. This PR simply adds that!

The driver will need to be updated to follow the convention for installed asset described in #1113 (section: "Assets for installed packages").

panglesd · 2024-08-01T13:32:29Z

(I'm not very proud of the name of the CLI option: --asset-path)

Julow · 2024-08-02T13:37:20Z

src/odoc/bin/main.ml

      let source = source_of_args source_root source_file in
      let file = Fs.File.of_string input_file in
      Rendering.generate_odoc ~renderer:R.renderer ~warnings_options ~syntax
-        ~output:output_dir ~extra_suffix ~source ~sidebar extra file
+        ~output:output_dir ~extra_suffix ~source ~sidebar ~asset_path extra file


I'm not a fan of adding more arguments to this command. This argument makes no sense in most cases and needs to be checked. Similarly, Document.Asset seems a bit out of place, as it interacts with nothing else, it could be out of the document thing entirely.

What about adding a different command for generating assets ? html-generate-asset, latex-generate-asset, etc..

The same is true for --source and --source-root, but OK, I'll do that!

(the Document.Asset will be inconvenient to remove in my opinion, unless we remove the old way of attaching assets, which would be a good thing I think)

panglesd · 2024-08-21T08:46:51Z

I have added support in the driver for the asset pipeline, following the convention.

This means, we now have images working for packages following the odig convention!

Julow

Looks good :)

src/driver/dune_style.ml

Julow · 2024-08-21T09:58:52Z

src/driver/opam.ml

@@ -135,6 +135,10 @@ let pkg_to_dir_map () =
              | "doc" :: _pkg :: "odoc-pages" :: _ ->
                  Logs.debug (fun m -> m "Found odoc page: %a" Fpath.pp fpath);

+                  (Fpath.Set.add Fpath.(v prefix // fpath) odoc_pages, others)
+              | "doc" :: _pkg :: "odoc-assets" :: _ ->


Unrelated to this PR but pages and assets are taken from the list of installed files for a package. I find that unnecessary and potentially buggy. A package could install doc in an other package or this file could be outdated (dune install), which might crash the driver.
I think the doc/pkgname folder should be scanned independently of pkg_contents. I think the same for libraries but there might be cases that require this?

Julow · 2024-08-21T10:12:02Z

There's a broken asset in vg.lib.vg.Vg.P because it's using an hardcoded path but I couldn't make the reference syntax work. Asset references from libraries might not work.

This removes a deprecated-before-use pipeline, and allows to finally remove the extra_documents in renderers

panglesd · 2024-08-22T15:42:58Z

This has been rebased on top of #1188. html-generate and html-generate-asset have been split. The old asset pipeline has been removed: --child asset-... for compile and --asset for html-generate.

Julow · 2024-08-22T16:43:01Z

src/document/renderer.ml

@@ -24,7 +24,6 @@ type input =
 type 'a t = {
  name : string;
  render : 'a -> Types.Block.t option -> Types.Document.t -> page list;
-  extra_documents : 'a -> input -> Types.Document.t list;


Julow · 2024-08-22T16:52:02Z

src/odoc/rendering.ml

+      let doc = Renderer.document_of_asset asset_file unit in
+      render_document renderer ~output ~sidebar:None ~extra_suffix ~extra doc;


There was a lot of simplification in this PR, why not continue and remove assets from the document and the renderer ?
document_of_asset seems very orthogonal to the rest of the document (it's just the Asset constructor which shares nothing with the other document types) and because of render_document and the renderer type, the content of the asset file has to go through a Format buffer, which seems totally unnecessary.
I think all of this could be removed.

True! (I started that and decided to stop, thinking you would not see it, so I stashed everything. Time to stash pop!)

Julow

I think the driver is broken on main as it calls odoc html-generate-impl but I could test this a bit.

src/odoc/fs.ml

src/driver/odoc.ml

Co-authored-by: Jules Aguillon <[email protected]>

panglesd · 2024-08-23T10:39:32Z

I think the driver is broken on main as it calls odoc html-generate-impl but I could test this a bit.

I'm being pragmatic and fixing this in this PR.

Julow

It's perfect now, let's merge !

@jonludlam

CHANGES: ### Highlight - Hierarchical documentation (@jonludlam, @panglesd, @Julow) Pages can now be organized in a directory tree structure. Relative and absolute references are added: `{!./other_page.label}`, `{!//other_page}`. - Improved sidebar and breadcrumbs navigation (@panglesd, @gpetiot) The documentation pages and the libraries of the entire package are shown on the left sidebar. - Added support for images, videos, audio and other assets The syntax is `{image!/reference/to/asset}` or `{image:URL}` for images. The syntax for `{video...}` and `{audio...}` is the same. (@panglesd, @EmileTrotignon, ocaml/odoc#1170, ocaml/odoc#1171, ocaml/odoc#1184, ocaml/odoc#1185) - Search using Sherlodoc (@panglesd, @EmileTrotignon, @Julow) A new search bar that supports full-text and type-based search. ### Added - Experimental driver (@jonludlam, @panglesd) The driver builds the documentation for a collection of Opam packages using the newer Odoc features. It supports linking external packages to ocaml.org and markdown files. This is experimental and will break in the future. - Cross-package references (@panglesd, @Julow) Pages and modules from other packages can be referenced: `{!/otherpackage/page}`, `{!/otherpackage/Module.t}`. - Option to remap links to other packages to ocaml.org or other site. See the `--remap` option of the driver or the `--remap-file` option of `odoc html-generate`. (@jonludlam, ocaml/odoc#1189, ocaml/odoc#1248) - Option to compute occurrences of use of each identifiers The commands `aggregate-occurrences` and `count-occurrences` are added. (@panglesd, ocaml/odoc#976, ocaml/odoc#1076, ocaml/odoc#1206) - Added the `odoc classify` command (@jonludlam, ocaml/odoc#1121) Helps driver detecting which modules belong to which libraries. - Added `--suppress-warnings` to the CLI to remove warnings from a unit, even if they end up being raised in another unit through expansion (@jonludlam, ocaml/odoc#1260) - Add clock emoji before `@since` tag (@yawaramin, ocaml/odoc#1089) - Navigation for the search bar : use '/' to enter search, up and down arrows to select a result, and enter to follow the selected link. (@EmileTrotignon, ocaml/odoc#1088) - Fix a big gap between the preamble and the content of a page (@EmileTrotignon, ocaml/odoc#1147) - Add a marshalled search index consumable by sherlodoc (@EmileTrotignon, @panglesd, ocaml/odoc#1084) - Allow referencing of polymorphic constructors in polymorphic variant type aliases (@panglesd, ocaml/odoc#1115) - Added a home icon in the breacrumbs (@panglesd, ocaml/odoc#1251) It can be disabled with a CLI option. - Add a frontmatter syntax for mld pages (@panglesd, ocaml/odoc#1187, ocaml/odoc#1193, ocaml/odoc#1243, ocaml/odoc#1246, ocaml/odoc#1251) Allows to specify the title of a page, the order of sub-pages and other behaviors in the sidebar. - Added `odoc-md` to process standalone Markdown pages (@jonludlam, ocaml/odoc#1234) ### Changed - The command line interface changed to support the new features. + Packages and libraries: `odoc link` must now be aware of packages and libraries with the `-L libname:path` and `-P pkgname:path` options. The module search path should still be passed with the `-I` option. The current package should be specified with `--current-package=pkgname`. + Hierarchy: `odoc compile` now outputs `.odoc` in the directory tree specified with `--output-dir=DIR` and the parent identifier must be specified with `--parent-id=PARENT`. The option `--source-parent-file` is removed. + Source code: Implementations are compiled with `compile-impl` instead of with `compile`. The options `--cmt=..` and `--source-name=..` are removed. Source code pages are generated with `html-generate-source`. + Assets: The commands `compile-asset`, `html-generate-asset` are added. The option `html-generate --asset` is removed. + Sidebar: The index is built using `compile-index`. The sidebar data is extracted from the index with `sidebar-generate` and passed to `html-generate --sidebar=..`. - The syntax for `@tag` is now delimited (@panglesd, ocaml/odoc#1239) A `@tag` can now be followed by a paragraph or other elements. - Updated colors for code fragments (@EmileTrotignon, ocaml/odoc#1023) - Fixed complexity of looking up `.odoc` files (@panglesd, ocaml/odoc#1075) - Normalize whitespaces in codespans (@gpetiot, ocaml/odoc#1085) A newline followed by any whitespaces is normalized as one space character. - Reduce size of `Odoc_html_frontend` when compiled to javascript (@EmileTrotignon, ocaml/odoc#1072) - Overhaul of module-type-of expansions and shadowing code (@jonludlam, ocaml/odoc#1081) - Output file paths and labels in the man and latex backends changed to avoid name clashes (@Julow, ocaml/odoc#1191) ### Fixed - Fix variant constructors being hidden if they contain hidden types (@jonludlam, ocaml/odoc#1105) - Fix rare assertion failure due to optional parameters (@jonludlam, ocaml/odoc#1272, issue ocaml/odoc#1001) - Fix resolution of module synopses in {!modules} lists that require --open (@jonludlam, ocaml/odoc#1104} - Fix top comment not being taken from includes often enough (@panglesd, ocaml/odoc#1117) - Fixed 404 links from search results (@panglesd, ocaml/odoc#1108) - Fixed title content not being picked up across pages when rendering references (ocaml/odoc#1116, @panglesd) - Fix wrong links to standalone comments in search results (ocaml/odoc#1118, @panglesd) - Remove duplicated or unwanted comments with inline includes (@Julow, ocaml/odoc#1133) - Fix bug where source rendering would cause odoc to fail completely if it encounters invalid syntax (@jonludlam ocaml/odoc#1208) - Add missing parentheses in 'val (let*) : ...' (@Julow, ocaml/odoc#1268) - Fix syntax highlighting not working for very large files (@jonludlam, @Julow, ocaml/odoc#1277)

@jonludlam

CHANGES: - Hierarchical documentation (@jonludlam, @panglesd, @Julow) Pages can now be organized in a directory tree structure. Relative and absolute references are added: `{!./other_page.label}`, `{!//other_page}`. - Improved sidebar and breadcrumbs navigation (@panglesd, @gpetiot) The documentation pages and the libraries of the entire package are shown on the left sidebar. - Added support for images, videos, audio and other assets The syntax is `{image!/reference/to/asset}` or `{image:URL}` for images. The syntax for `{video...}` and `{audio...}` is the same. (@panglesd, @EmileTrotignon, ocaml/odoc#1170, ocaml/odoc#1171, ocaml/odoc#1184, ocaml/odoc#1185) - Search using Sherlodoc (@panglesd, @EmileTrotignon, @Julow) A new search bar that supports full-text and type-based search. - Experimental driver (@jonludlam, @panglesd) The driver builds the documentation for a collection of Opam packages using the newer Odoc features. It supports linking external packages to ocaml.org and markdown files. This is experimental and will break in the future. - Cross-package references (@panglesd, @Julow) Pages and modules from other packages can be referenced: `{!/otherpackage/page}`, `{!/otherpackage/Module.t}`. - Option to remap links to other packages to ocaml.org or other site. See the `--remap` option of the driver or the `--remap-file` option of `odoc html-generate`. (@jonludlam, ocaml/odoc#1189, ocaml/odoc#1248) - Option to compute occurrences of use of each identifiers The commands `aggregate-occurrences` and `count-occurrences` are added. (@panglesd, ocaml/odoc#976, ocaml/odoc#1076, ocaml/odoc#1206) - Added the `odoc classify` command (@jonludlam, ocaml/odoc#1121) Helps driver detecting which modules belong to which libraries. - Added `--suppress-warnings` to the CLI to remove warnings from a unit, even if they end up being raised in another unit through expansion (@jonludlam, ocaml/odoc#1260) - Add clock emoji before `@since` tag (@yawaramin, ocaml/odoc#1089) - Navigation for the search bar : use '/' to enter search, up and down arrows to select a result, and enter to follow the selected link. (@EmileTrotignon, ocaml/odoc#1088) - Fix a big gap between the preamble and the content of a page (@EmileTrotignon, ocaml/odoc#1147) - Add a marshalled search index consumable by sherlodoc (@EmileTrotignon, @panglesd, ocaml/odoc#1084) - Allow referencing of polymorphic constructors in polymorphic variant type aliases (@panglesd, ocaml/odoc#1115) - Added a home icon in the breacrumbs (@panglesd, ocaml/odoc#1251) It can be disabled with a CLI option. - Add a frontmatter syntax for mld pages (@panglesd, ocaml/odoc#1187, ocaml/odoc#1193, ocaml/odoc#1243, ocaml/odoc#1246, ocaml/odoc#1251) Allows to specify the title of a page, the order of sub-pages and other behaviors in the sidebar. - Added `odoc-md` to process standalone Markdown pages (@jonludlam, ocaml/odoc#1234) - The command line interface changed to support the new features. + Packages and libraries: `odoc link` must now be aware of packages and libraries with the `-L libname:path` and `-P pkgname:path` options. The module search path should still be passed with the `-I` option. The current package should be specified with `--current-package=pkgname`. + Hierarchy: `odoc compile` now outputs `.odoc` in the directory tree specified with `--output-dir=DIR` and the parent identifier must be specified with `--parent-id=PARENT`. The option `--source-parent-file` is removed. + Source code: Implementations are compiled with `compile-impl` instead of with `compile`. The options `--cmt=..` and `--source-name=..` are removed. Source code pages are generated with `html-generate-source`. + Assets: The commands `compile-asset`, `html-generate-asset` are added. The option `html-generate --asset` is removed. + Sidebar: The index is built using `compile-index`. The sidebar data is extracted from the index with `sidebar-generate` and passed to `html-generate --sidebar=..`. - The syntax for `@tag` is now delimited (@panglesd, ocaml/odoc#1239) A `@tag` can now be followed by a paragraph or other elements. - Updated colors for code fragments (@EmileTrotignon, ocaml/odoc#1023) - Fixed complexity of looking up `.odoc` files (@panglesd, ocaml/odoc#1075) - Normalize whitespaces in codespans (@gpetiot, ocaml/odoc#1085) A newline followed by any whitespaces is normalized as one space character. - Reduce size of `Odoc_html_frontend` when compiled to javascript (@EmileTrotignon, ocaml/odoc#1072) - Overhaul of module-type-of expansions and shadowing code (@jonludlam, ocaml/odoc#1081) - Output file paths and labels in the man and latex backends changed to avoid name clashes (@Julow, ocaml/odoc#1191) - Fix variant constructors being hidden if they contain hidden types (@jonludlam, ocaml/odoc#1105) - Fix rare assertion failure due to optional parameters (@jonludlam, ocaml/odoc#1272, issue ocaml/odoc#1001) - Fix resolution of module synopses in {!modules} lists that require --open (@jonludlam, ocaml/odoc#1104} - Fix top comment not being taken from includes often enough (@panglesd, ocaml/odoc#1117) - Fixed 404 links from search results (@panglesd, ocaml/odoc#1108) - Fixed title content not being picked up across pages when rendering references (ocaml/odoc#1116, @panglesd) - Fix wrong links to standalone comments in search results (ocaml/odoc#1118, @panglesd) - Remove duplicated or unwanted comments with inline includes (@Julow, ocaml/odoc#1133) - Fix bug where source rendering would cause odoc to fail completely if it encounters invalid syntax (@jonludlam ocaml/odoc#1208) - Add missing parentheses in 'val (let*) : ...' (@Julow, ocaml/odoc#1268) - Fix syntax highlighting not working for very large files (@jonludlam, @Julow, ocaml/odoc#1277)

panglesd added 2 commits August 1, 2024 15:47

Add ability to call generate commands on asset units

fc52c23

Added changelog for asset generation

5741d4b

panglesd force-pushed the generate-assets branch from 2de1814 to 5741d4b Compare August 1, 2024 13:47

Julow reviewed Aug 2, 2024

View reviewed changes

panglesd mentioned this pull request Aug 20, 2024

Separate commands for html generation of page+units and implementation #1188

Merged

panglesd added 2 commits August 21, 2024 10:30

Driver: Compile assets, following convention

3f69941

Driver: add a counter for compiling assets

c185847

Julow approved these changes Aug 21, 2024

View reviewed changes

panglesd added 6 commits August 21, 2024 13:42

Driver: clarify comment on assets in dune style

65be5a3

Merge branch 'master' into generate-assets

e395d1a

Separate html-generate and html-generate-asset

1e9cf37

Remove the old asset pipeline

f04cce9

This removes a deprecated-before-use pipeline, and allows to finally remove the extra_documents in renderers

Driver: fix assets

996e661

Fix unpromoted test

51cd330

Julow reviewed Aug 22, 2024

View reviewed changes

panglesd added 5 commits August 23, 2024 10:53

Fully remove Assets from the document abstraction

18cb404

Asset generation: factor out code

3f9277d

Compatibility

1f8e833

More annoying compatibility degrading the experience for everyone

1e2d55a

Fix docv for asset unit file

f8226f7

Julow reviewed Aug 23, 2024

View reviewed changes

src/odoc/fs.ml Outdated Show resolved Hide resolved

src/driver/odoc.ml Outdated Show resolved Hide resolved

panglesd and others added 3 commits August 23, 2024 12:16

Fix util copy function

c581d4e

Driver: update to new generating asset CLI

17b20ba

Co-authored-by: Jules Aguillon <[email protected]>

Driver: fix command for building source

2444176

Julow approved these changes Aug 23, 2024

View reviewed changes

panglesd merged commit d6fea07 into ocaml:master Aug 23, 2024
12 checks passed

panglesd mentioned this pull request Aug 23, 2024

Media in odoc 3! #1184

Merged

jonludlam mentioned this pull request Jan 23, 2025

[new release] odoc (5 packages) (3.0.0~beta1) ocaml/opam-repository#27309

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ability to call generate commands on asset units #1185

Add ability to call generate commands on asset units #1185

panglesd commented Aug 1, 2024

panglesd commented Aug 1, 2024

Julow Aug 2, 2024 •

edited

Loading

panglesd Aug 19, 2024

panglesd Aug 19, 2024

panglesd commented Aug 21, 2024

Julow left a comment

Julow Aug 21, 2024

Julow commented Aug 21, 2024

panglesd commented Aug 22, 2024

Julow Aug 22, 2024

Julow Aug 22, 2024

panglesd Aug 22, 2024

Julow left a comment

panglesd commented Aug 23, 2024

Julow left a comment

		let doc = Renderer.document_of_asset asset_file unit in
		render_document renderer ~output ~sidebar:None ~extra_suffix ~extra doc;

Add ability to call generate commands on asset units #1185

Add ability to call generate commands on asset units #1185

Conversation

panglesd commented Aug 1, 2024

panglesd commented Aug 1, 2024

Julow Aug 2, 2024 • edited Loading

Choose a reason for hiding this comment

panglesd Aug 19, 2024

Choose a reason for hiding this comment

panglesd Aug 19, 2024

Choose a reason for hiding this comment

panglesd commented Aug 21, 2024

Julow left a comment

Choose a reason for hiding this comment

Julow Aug 21, 2024

Choose a reason for hiding this comment

Julow commented Aug 21, 2024

panglesd commented Aug 22, 2024

Julow Aug 22, 2024

Choose a reason for hiding this comment

Julow Aug 22, 2024

Choose a reason for hiding this comment

panglesd Aug 22, 2024

Choose a reason for hiding this comment

Julow left a comment

Choose a reason for hiding this comment

panglesd commented Aug 23, 2024

Julow left a comment

Choose a reason for hiding this comment

Julow Aug 2, 2024 •

edited

Loading