Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Find odoc files in accessible path more efficiently #1075

Merged
merged 6 commits into from
Feb 13, 2024

Conversation

panglesd
Copy link
Collaborator

Previously, when $n$ -I arguments were given, looking for a .odoc file was of complexity $n$, so looking for $n$ files was of complexity $n^2$, which could take 99.99% of a compile command in some extreme case (when a lot of -I are given to the compile command).

This commit turns that in $O(n)$ where $n$ is the number of files available in all directories, by doing an "ls" in each directory and caching the result.

So, it performs slightly worse in the case of a single -I with many files that are not accessed .odoc in it, but drastically better in case of many -I with (accessed) .odoc files in them.

panglesd added a commit to panglesd/odoc that referenced this pull request Jan 29, 2024
@panglesd
Copy link
Collaborator Author

There is a TODO which I think could be fixed in another PR: what to do in case of an inexistent/unavailable directory given to -I.

There was no warning, and it is still the case, but at some point it would be good to emit a warning.

Copy link
Collaborator

@Julow Julow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The benchmark don't seem to contain a reproduction for the mentioned bug but the numbers look ok: compile time increased slightly and link time decreased by about the same (+0.003s per compile, -0.006s per link).

The code looks good.

@jonludlam
Copy link
Member

Could we please have a test case that demonstrates the problem and the fix?

panglesd added a commit to panglesd/odoc that referenced this pull request Feb 12, 2024
@panglesd panglesd force-pushed the squared-search-path branch from 5ca08fc to cca1f90 Compare February 12, 2024 14:04
panglesd added a commit to panglesd/odoc that referenced this pull request Feb 12, 2024
@panglesd panglesd force-pushed the squared-search-path branch from cca1f90 to ba6da6c Compare February 12, 2024 14:05
@panglesd
Copy link
Collaborator Author

panglesd commented Feb 12, 2024

I added a test with 1000 -I and 1000 different lookups, so 1000000 call to Unix.stat (using the old mechanism).
It used to take more than 5s to run, now it takes less than 0.1s.

I'll try to investigate the situation where it is worse: one single -I with many unrelated files in it.

panglesd added a commit to panglesd/odoc that referenced this pull request Feb 12, 2024
@panglesd panglesd force-pushed the squared-search-path branch from ba6da6c to d1b348f Compare February 12, 2024 14:25
See next commit for more explanation

Signed-off-by: Paul-Elliot <[email protected]>
Previously, when $n$ `-I` arguments were given, looking for a `.odoc` file was
of complexity $n$, so looking for $n$ files was of complexity $n^2$, which could
take 99.99% of a compile command in some extreme case.

This commit turns that in $O(n)$ where $n$ is the number of files available in
all directories, by doing an "`ls`" in each directory and caching the result.

It performs slightly worse in the case of a single `-I` with many files that are
not accessed `.odoc` in it, but drastically better in case of many `-I` with
mostly accessed `.odoc` files in them.

Signed-off-by: Paul-Elliot <[email protected]>
@panglesd panglesd force-pushed the squared-search-path branch from d1b348f to 08aad1a Compare February 12, 2024 14:41
@panglesd panglesd force-pushed the squared-search-path branch from 8a10446 to 4ab4a70 Compare February 12, 2024 15:15
@panglesd panglesd force-pushed the squared-search-path branch from 4ab4a70 to 5e1bfe8 Compare February 12, 2024 15:36
@panglesd
Copy link
Collaborator Author

@jonludlam I added a test for the case that is now worse, and it seems to be OK.

@jonludlam
Copy link
Member

timeout doesn't exist on my mac. Also, some of the tests are failing because 0.1 seconds is too aggressive! :-D

A more portable timeout-like thing could be:

(./slow_odoc_invocation.sh; echo 'done')  & sleep 1; kill $! 2> /dev/null || true

see e.g. http://blog.mediatribe.net/en/node/72/index.html

@panglesd panglesd force-pushed the squared-search-path branch 2 times, most recently from 8fc5936 to b93c46b Compare February 13, 2024 08:32
@panglesd
Copy link
Collaborator Author

Thanks for the mac trick.

The failing CI does not seem to be relevant for this PR:

[ERROR] The compilation of jst-config.v0.16.0 failed at "dune build -p jst-config -j 3".

@jonludlam
Copy link
Member

great thanks!

@jonludlam jonludlam merged commit 7085a7c into ocaml:master Feb 13, 2024
9 of 10 checks passed
jonludlam pushed a commit that referenced this pull request Feb 13, 2024
jonludlam added a commit to jonludlam/opam-repository that referenced this pull request Jan 23, 2025
CHANGES:

### Highlight

- Hierarchical documentation (@jonludlam, @panglesd, @Julow)
  Pages can now be organized in a directory tree structure.
  Relative and absolute references are added:
  `{!./other_page.label}`, `{!//other_page}`.

- Improved sidebar and breadcrumbs navigation (@panglesd, @gpetiot)
  The documentation pages and the libraries of the entire package are shown on
  the left sidebar.

- Added support for images, videos, audio and other assets
  The syntax is `{image!/reference/to/asset}` or `{image:URL}` for images.
  The syntax for `{video...}` and `{audio...}` is the same.
  (@panglesd, @EmileTrotignon, ocaml/odoc#1170, ocaml/odoc#1171, ocaml/odoc#1184, ocaml/odoc#1185)

- Search using Sherlodoc (@panglesd, @EmileTrotignon, @Julow)
  A new search bar that supports full-text and type-based search.

### Added

- Experimental driver (@jonludlam, @panglesd)
  The driver builds the documentation for a collection of Opam packages using
  the newer Odoc features. It supports linking external packages to ocaml.org
  and markdown files.
  This is experimental and will break in the future.

- Cross-package references (@panglesd, @Julow)
  Pages and modules from other packages can be referenced:
  `{!/otherpackage/page}`, `{!/otherpackage/Module.t}`.

- Option to remap links to other packages to ocaml.org or other site.
  See the `--remap` option of the driver or the `--remap-file` option of `odoc html-generate`.
  (@jonludlam, ocaml/odoc#1189, ocaml/odoc#1248)

- Option to compute occurrences of use of each identifiers
  The commands `aggregate-occurrences` and `count-occurrences` are added.
  (@panglesd, ocaml/odoc#976, ocaml/odoc#1076, ocaml/odoc#1206)

- Added the `odoc classify` command (@jonludlam, ocaml/odoc#1121)
  Helps driver detecting which modules belong to which libraries.
- Added `--suppress-warnings` to the CLI to remove warnings from a unit, even
  if they end up being raised in another unit through expansion
  (@jonludlam, ocaml/odoc#1260)
- Add clock emoji before `@since` tag (@yawaramin, ocaml/odoc#1089)
- Navigation for the search bar : use '/' to enter search, up and down arrows to
  select a result, and enter to follow the selected link. (@EmileTrotignon, ocaml/odoc#1088)
- Fix a big gap between the preamble and the content of a page (@EmileTrotignon, ocaml/odoc#1147)
- Add a marshalled search index consumable by sherlodoc (@EmileTrotignon, @panglesd, ocaml/odoc#1084)
- Allow referencing of polymorphic constructors in polymorphic variant type
  aliases (@panglesd, ocaml/odoc#1115)
- Added a home icon in the breacrumbs (@panglesd, ocaml/odoc#1251)
  It can be disabled with a CLI option.
- Add a frontmatter syntax for mld pages (@panglesd, ocaml/odoc#1187, ocaml/odoc#1193, ocaml/odoc#1243, ocaml/odoc#1246, ocaml/odoc#1251)
  Allows to specify the title of a page, the order of sub-pages and other
  behaviors in the sidebar.
- Added `odoc-md` to process standalone Markdown pages (@jonludlam, ocaml/odoc#1234)

### Changed

- The command line interface changed to support the new features.
  + Packages and libraries: `odoc link` must now be aware of packages and
    libraries with the `-L libname:path` and `-P pkgname:path` options. The
    module search path should still be passed with the `-I` option.
    The current package should be specified with `--current-package=pkgname`.
  + Hierarchy: `odoc compile` now outputs `.odoc` in the directory tree
    specified with `--output-dir=DIR` and the parent identifier must be
    specified with `--parent-id=PARENT`.
    The option `--source-parent-file` is removed.
  + Source code: Implementations are compiled with `compile-impl` instead of
    with `compile`. The options `--cmt=..` and `--source-name=..` are removed.
    Source code pages are generated with `html-generate-source`.
  + Assets: The commands `compile-asset`, `html-generate-asset` are added.
    The option `html-generate --asset` is removed.
  + Sidebar: The index is built using `compile-index`. The sidebar data is
    extracted from the index with `sidebar-generate` and passed to
    `html-generate --sidebar=..`.

- The syntax for `@tag` is now delimited (@panglesd, ocaml/odoc#1239)
  A `@tag` can now be followed by a paragraph or other elements.

- Updated colors for code fragments (@EmileTrotignon, ocaml/odoc#1023)
- Fixed complexity of looking up `.odoc` files (@panglesd, ocaml/odoc#1075)
- Normalize whitespaces in codespans (@gpetiot, ocaml/odoc#1085)
  A newline followed by any whitespaces is normalized as one space character.
- Reduce size of `Odoc_html_frontend` when compiled to javascript
  (@EmileTrotignon, ocaml/odoc#1072)
- Overhaul of module-type-of expansions and shadowing code (@jonludlam, ocaml/odoc#1081)
- Output file paths and labels in the man and latex backends changed to avoid
  name clashes (@Julow, ocaml/odoc#1191)

### Fixed

- Fix variant constructors being hidden if they contain hidden types
  (@jonludlam, ocaml/odoc#1105)
- Fix rare assertion failure due to optional parameters
  (@jonludlam, ocaml/odoc#1272, issue ocaml/odoc#1001)
- Fix resolution of module synopses in {!modules} lists that require --open
  (@jonludlam, ocaml/odoc#1104}
- Fix top comment not being taken from includes often enough (@panglesd, ocaml/odoc#1117)
- Fixed 404 links from search results (@panglesd, ocaml/odoc#1108)
- Fixed title content not being picked up across pages when rendering references
  (ocaml/odoc#1116, @panglesd)
- Fix wrong links to standalone comments in search results (ocaml/odoc#1118, @panglesd)
- Remove duplicated or unwanted comments with inline includes (@Julow, ocaml/odoc#1133)
- Fix bug where source rendering would cause odoc to fail completely if it
  encounters invalid syntax (@jonludlam ocaml/odoc#1208)
- Add missing parentheses in 'val (let*) : ...' (@Julow, ocaml/odoc#1268)
- Fix syntax highlighting not working for very large files
  (@jonludlam, @Julow, ocaml/odoc#1277)
jonludlam added a commit to jonludlam/opam-repository that referenced this pull request Jan 23, 2025
CHANGES:

- Hierarchical documentation (@jonludlam, @panglesd, @Julow)
  Pages can now be organized in a directory tree structure.
  Relative and absolute references are added:
  `{!./other_page.label}`, `{!//other_page}`.

- Improved sidebar and breadcrumbs navigation (@panglesd, @gpetiot)
  The documentation pages and the libraries of the entire package are shown on
  the left sidebar.

- Added support for images, videos, audio and other assets
  The syntax is `{image!/reference/to/asset}` or `{image:URL}` for images.
  The syntax for `{video...}` and `{audio...}` is the same.
  (@panglesd, @EmileTrotignon, ocaml/odoc#1170, ocaml/odoc#1171, ocaml/odoc#1184, ocaml/odoc#1185)

- Search using Sherlodoc (@panglesd, @EmileTrotignon, @Julow)
  A new search bar that supports full-text and type-based search.

- Experimental driver (@jonludlam, @panglesd)
  The driver builds the documentation for a collection of Opam packages using
  the newer Odoc features. It supports linking external packages to ocaml.org
  and markdown files.
  This is experimental and will break in the future.

- Cross-package references (@panglesd, @Julow)
  Pages and modules from other packages can be referenced:
  `{!/otherpackage/page}`, `{!/otherpackage/Module.t}`.

- Option to remap links to other packages to ocaml.org or other site.
  See the `--remap` option of the driver or the `--remap-file` option of `odoc html-generate`.
  (@jonludlam, ocaml/odoc#1189, ocaml/odoc#1248)

- Option to compute occurrences of use of each identifiers
  The commands `aggregate-occurrences` and `count-occurrences` are added.
  (@panglesd, ocaml/odoc#976, ocaml/odoc#1076, ocaml/odoc#1206)

- Added the `odoc classify` command (@jonludlam, ocaml/odoc#1121)
  Helps driver detecting which modules belong to which libraries.
- Added `--suppress-warnings` to the CLI to remove warnings from a unit, even
  if they end up being raised in another unit through expansion
  (@jonludlam, ocaml/odoc#1260)
- Add clock emoji before `@since` tag (@yawaramin, ocaml/odoc#1089)
- Navigation for the search bar : use '/' to enter search, up and down arrows to
  select a result, and enter to follow the selected link. (@EmileTrotignon, ocaml/odoc#1088)
- Fix a big gap between the preamble and the content of a page (@EmileTrotignon, ocaml/odoc#1147)
- Add a marshalled search index consumable by sherlodoc (@EmileTrotignon, @panglesd, ocaml/odoc#1084)
- Allow referencing of polymorphic constructors in polymorphic variant type
  aliases (@panglesd, ocaml/odoc#1115)
- Added a home icon in the breacrumbs (@panglesd, ocaml/odoc#1251)
  It can be disabled with a CLI option.
- Add a frontmatter syntax for mld pages (@panglesd, ocaml/odoc#1187, ocaml/odoc#1193, ocaml/odoc#1243, ocaml/odoc#1246, ocaml/odoc#1251)
  Allows to specify the title of a page, the order of sub-pages and other
  behaviors in the sidebar.
- Added `odoc-md` to process standalone Markdown pages (@jonludlam, ocaml/odoc#1234)

- The command line interface changed to support the new features.
  + Packages and libraries: `odoc link` must now be aware of packages and
    libraries with the `-L libname:path` and `-P pkgname:path` options. The
    module search path should still be passed with the `-I` option.
    The current package should be specified with `--current-package=pkgname`.
  + Hierarchy: `odoc compile` now outputs `.odoc` in the directory tree
    specified with `--output-dir=DIR` and the parent identifier must be
    specified with `--parent-id=PARENT`.
    The option `--source-parent-file` is removed.
  + Source code: Implementations are compiled with `compile-impl` instead of
    with `compile`. The options `--cmt=..` and `--source-name=..` are removed.
    Source code pages are generated with `html-generate-source`.
  + Assets: The commands `compile-asset`, `html-generate-asset` are added.
    The option `html-generate --asset` is removed.
  + Sidebar: The index is built using `compile-index`. The sidebar data is
    extracted from the index with `sidebar-generate` and passed to
    `html-generate --sidebar=..`.

- The syntax for `@tag` is now delimited (@panglesd, ocaml/odoc#1239)
  A `@tag` can now be followed by a paragraph or other elements.

- Updated colors for code fragments (@EmileTrotignon, ocaml/odoc#1023)
- Fixed complexity of looking up `.odoc` files (@panglesd, ocaml/odoc#1075)
- Normalize whitespaces in codespans (@gpetiot, ocaml/odoc#1085)
  A newline followed by any whitespaces is normalized as one space character.
- Reduce size of `Odoc_html_frontend` when compiled to javascript
  (@EmileTrotignon, ocaml/odoc#1072)
- Overhaul of module-type-of expansions and shadowing code (@jonludlam, ocaml/odoc#1081)
- Output file paths and labels in the man and latex backends changed to avoid
  name clashes (@Julow, ocaml/odoc#1191)

- Fix variant constructors being hidden if they contain hidden types
  (@jonludlam, ocaml/odoc#1105)
- Fix rare assertion failure due to optional parameters
  (@jonludlam, ocaml/odoc#1272, issue ocaml/odoc#1001)
- Fix resolution of module synopses in {!modules} lists that require --open
  (@jonludlam, ocaml/odoc#1104}
- Fix top comment not being taken from includes often enough (@panglesd, ocaml/odoc#1117)
- Fixed 404 links from search results (@panglesd, ocaml/odoc#1108)
- Fixed title content not being picked up across pages when rendering references
  (ocaml/odoc#1116, @panglesd)
- Fix wrong links to standalone comments in search results (ocaml/odoc#1118, @panglesd)
- Remove duplicated or unwanted comments with inline includes (@Julow, ocaml/odoc#1133)
- Fix bug where source rendering would cause odoc to fail completely if it
  encounters invalid syntax (@jonludlam ocaml/odoc#1208)
- Add missing parentheses in 'val (let*) : ...' (@Julow, ocaml/odoc#1268)
- Fix syntax highlighting not working for very large files
  (@jonludlam, @Julow, ocaml/odoc#1277)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants