[libc][docs] Update docs to reflect new headergen (#102381)

Since new headergen is now the default for building LLVM-libc, the docs
need to be updated to reflect that. While I was editing those docs, I
took a quick pass at updating other out-of-date pages.

Author: michaelrj-google

libc/docs/build_and_test.rst

@@ -38,13 +38,6 @@ The libc can be built and tested in two different modes:
 
         $> ninja libc-integration-tests
 
-   #. API verification test - See :ref:`api_test` for more information about
-      the API test. It can be run by the command:
-
-      .. code-block:: sh
-
-        $> ninja libc-api-test
-
 Building with VSCode
 ====================
 

libc/docs/contributing.rst

@@ -4,7 +4,7 @@
 Contributing to the libc Project
 ================================
 
-LLVM's libc is being developed as part of the LLVM project so contributions
+LLVM-libc is being developed as part of the LLVM project so contributions
 to the libc project should also follow the general LLVM
 `contribution guidelines <https://llvm.org/docs/Contributing.html>`_. Below is
 a list of open projects that one can start with:
@@ -31,24 +31,12 @@ a list of open projects that one can start with:
    directory. So, a simple but mechanical project would be to move the parts
    following the old styles to the new style.
 
-#. **Integrating with the rest of the LLVM project** - There are two parts to
-   this project:
-
-   #. One is about adding CMake facilities to optionally link the libc's overlay
-      static archive (see :ref:`overlay_mode`) with other LLVM tools/executables.
-   #. The other is about putting plumbing in place to release the overlay static
-      archive (see :ref:`overlay_mode`) as part of the LLVM binary releases.
-
 #. **Implement Linux syscall wrappers** - A large portion of the POSIX API can
    be implemented as syscall wrappers on Linux. A good number have already been
    implemented but many more are yet to be implemented. So, a project of medium
    complexity would be to implement syscall wrappers which have not yet been
    implemented.
 
-#. **Add a better random number generator** - The current random number
-   generator has a very small range. This has to be improved or switched over
-   to a fast random number generator with a large range.
-
 #. **Update the clang-tidy lint rules and use them in the build and/or CI** -
    Currently, the :ref:`clang_tidy_checks` have gone stale and are mostly unused
    by the developers and on the CI builders. This project is about updating

libc/docs/dev/api_test.rst

@@ -1,3 +1,5 @@
+.. _header_generation:
+
 Generating Public and Internal headers
 ======================================
 

libc/docs/dev/ground_truth_specification.rst

@@ -15,10 +15,7 @@ Navigate to the links below for information on the respective topics:
    config_options
    clang_tidy_checks
    fuzzing
-   ground_truth_specification
    header_generation
    implementation_standard
    undefined_behavior
    printf_behavior
-   api_test
-   mechanics_of_public_api

libc/docs/dev/header_generation.rst

@@ -14,9 +14,10 @@ directories::
         - docs
         - examples
         - fuzzing
+        - hdr
         - include
         - lib
-        - spec
+        - newhdrgen
         - src
         - startup
         - test
@@ -62,6 +63,14 @@ The directory structure within this directory mirrors the directory structure
 of the top-level ``libc`` directory itself. For more details, see
 :doc:`fuzzing`.
 
+The ``hdr`` directory
+---------------------
+
+This directory contains proxy headers which are included from the files in the
+src directory. These proxy headers either include our internal type or macro
+definitions, or the system's type or macro definitions, depending on if we are
+in fullbuild or overlay mode.
+
 The ``include`` directory
 -------------------------
 
@@ -80,13 +89,14 @@ The ``lib`` directory
 This directory contains a ``CMakeLists.txt`` file listing the targets for the
 public libraries ``libc.a``, ``libm.a`` etc.
 
-The ``spec`` directory
-----------------------
+The ``newhdrgen`` directory
+---------------------------
 
-This directory contains the specifications for the types, macros, and entrypoint
-functions. These definitions come from the various standards and extensions
-LLVM-libc supports, and they are used along with the ``*.h.def`` files and the
-config files to generate the headers for fullbuild mode.
+This directory contains the sources and specifications for the types, macros
+and entrypoint functions. These definitions are organized in the ``yaml``
+subdirectory and match the organization of the ``*.h.def`` files. This folder
+also contains the python sources for new headergen, which is what generates the
+headers.
 
 The ``src`` directory
 ---------------------

libc/docs/dev/index.rst

@@ -8,35 +8,33 @@ Full Cross Build
    :depth: 1
    :local:
 
+.. note:: 
+   Fullbuild requires running headergen, which is a python program that depends on
+   pyyaml. The minimum versions are listed on the :ref:`header_generation`
+   page, as well as additional information.
+
 In this document, we will present recipes to cross build the full libc. When we
 say *cross build* a full libc, we mean that we will build the full libc for a
 target system which is not the same as the system on which the libc is being
 built. For example, you could be building for a bare metal aarch64 *target* on a
 Linux x86_64 *host*.
 
-There are three main recipes to cross build the full libc. Each one serves a
+There are two main recipes to cross build the full libc. Each one serves a
 different use case. Below is a short description of these recipes to help users
 pick the recipe that best suites their needs and contexts.
 
 * **Standalone cross build** - Using this recipe one can build the libc using a
   compiler of their choice. One should use this recipe if their compiler can
   build for the host as well as the target.
-* **Runtimes cross build** - In this recipe, one will have to first build the
-  libc build tools for the host separately and then use those build tools to
-  build the libc. Users can use the compiler of their choice to build the
-  libc build tools as well as the libc. One should use this recipe if they
-  have to use a host compiler to build the build tools for the host and then
-  use a target compiler (which is different from the host compiler) to build
-  the libc.
 * **Bootstrap cross build** - In this recipe, one will build the ``clang``
   compiler and the libc build tools for the host first, and then use them to
-  build the libc for the target. Unlike with the runtimes build recipe, the
-  user does not have explicitly build ``clang`` and other libc build tools.
+  build the libc for the target. Unlike with the standalone build recipe, the
+  user does not have explicitly build ``clang`` and other build tools.
   They get built automatically before building the libc. One should use this
   recipe if they intend use the built ``clang`` and the libc as part of their
   toolchain for the target.
 
-The following sections present the three recipes in detail.
+The following sections present the two recipes in detail.
 
 Standalone cross build
 ======================
@@ -61,9 +59,9 @@ Below is the CMake command to configure the standalone crossbuild of the libc.
   $> cd build
   $> C_COMPILER=<C compiler> # For example "clang"
   $> CXX_COMPILER=<C++ compiler> # For example "clang++"
-  $> cmake ../llvm  \
+  $> cmake ../runtimes  \
      -G Ninja \
-     -DLLVM_ENABLE_PROJECTS=libc  \
+     -DLLVM_ENABLE_RUNTIMES=libc  \
      -DCMAKE_C_COMPILER=$C_COMPILER \
      -DCMAKE_CXX_COMPILER=$CXX_COMPILER \
      -DLLVM_LIBC_FULL_BUILD=ON \
@@ -72,8 +70,8 @@ Below is the CMake command to configure the standalone crossbuild of the libc.
 
 We will go over the special options passed to the ``cmake`` command above.
 
-* **Enabled Projects** - Since we want to build the libc project, we list
-  ``libc`` as the enabled project.
+* **Enabled Runtimes** - Since we want to build LLVM-libc, we list
+  ``libc`` as the enabled runtime.
 * **The full build option** - Since we want to build the full libc, we pass
   ``-DLLVM_LIBC_FULL_BUILD=ON``.
 * **The target triple** - This is the target triple of the target for which
@@ -94,88 +92,6 @@ The above ``ninja`` command will build the libc static archives ``libc.a`` and
 ``libm.a`` for the target specified with ``-DLIBC_TARGET_TRIPLE`` in the CMake
 configure step.
 
-.. _runtimes_cross_build:
-
-Runtimes cross build
-====================
-
-The *runtimes cross build* is very similar to the standalone crossbuild but the
-user will have to first build the libc build tools for the host separately. One
-should use this recipe if they want to use a different host and target compiler.
-Note that the libc build tools MUST be in sync with the libc. That is, the
-libc build tools and the libc, both should be built from the same source
-revision. At the time of this writing, there is only one libc build tool that
-has to be built separately. It is done as follows:
-
-.. code-block:: sh
-
-  $> cd llvm-project  # The llvm-project checkout
-  $> mkdir build-libc-tools # A different build directory for the build tools
-  $> cd build-libc-tools
-  $> HOST_C_COMPILER=<C compiler for the host> # For example "clang"
-  $> HOST_CXX_COMPILER=<C++ compiler for the host> # For example "clang++"
-  $> cmake ../llvm  \
-     -G Ninja \
-     -DLLVM_ENABLE_PROJECTS=libc  \
-     -DCMAKE_C_COMPILER=$HOST_C_COMPILER \
-     -DCMAKE_CXX_COMPILER=$HOST_CXX_COMPILER  \
-     -DLLVM_LIBC_FULL_BUILD=ON \
-     -DCMAKE_BUILD_TYPE=Debug # User can choose to use "Release" build type
-  $> ninja libc-hdrgen
-
-The above commands should build a binary named ``libc-hdrgen``. Copy this binary
-to a directory of your choice.
-
-CMake configure step
---------------------
-
-After copying the ``libc-hdrgen`` binary to say ``/path/to/libc-hdrgen``,
-configure the libc build using the following command:
-
-.. code-block:: sh
-
-  $> cd llvm-project  # The llvm-project checkout
-  $> mkdir build
-  $> cd build
-  $> TARGET_C_COMPILER=<C compiler for the target>
-  $> TARGET_CXX_COMPILER=<C++ compiler for the target>
-  $> HDRGEN=</path/to/libc-hdrgen>
-  $> TARGET_TRIPLE=<Your target triple>
-  $> cmake ../runtimes  \
-     -G Ninja \
-     -DLLVM_ENABLE_RUNTIMES=libc  \
-     -DCMAKE_C_COMPILER=$TARGET_C_COMPILER \
-     -DCMAKE_CXX_COMPILER=$TARGET_CXX_COMPILER \
-     -DLLVM_LIBC_FULL_BUILD=ON \
-     -DLIBC_HDRGEN_EXE=$HDRGEN \
-     -DLIBC_TARGET_TRIPLE=$TARGET_TRIPLE \
-     -DCMAKE_BUILD_TYPE=Debug # User can choose to use "Release" build type
-
-Note the differences in the above cmake command versus the one used in the
-CMake configure step of the standalone build recipe:
-
-* Instead of listing ``libc`` in ``LLVM_ENABLED_PROJECTS``, we list it in
-  ``LLVM_ENABLED_RUNTIMES``.
-* Instead of using ``llvm-project/llvm`` as the root CMake source directory,
-  we use ``llvm-project/runtimes`` as the root CMake source directory.
-* The path to the ``libc-hdrgen`` binary built earlier is specified with
-  ``-DLIBC_HDRGEN_EXE=/path/to/libc-hdrgen``.
-
-Build step
-----------
-
-The build step in the runtimes build recipe is exactly the same as that of
-the standalone build recipe:
-
-.. code-block:: sh
-
-    $> ninja libc libm
-
-As with the standalone build recipe, the above ninja command will build the
-libc static archives for the target specified with ``-DLIBC_TARGET_TRIPLE`` in
-the CMake configure step.
-
-
 Bootstrap cross build
 =====================
 
@@ -203,8 +119,7 @@ CMake configure step
      -DLLVM_RUNTIME_TARGETS=$TARGET_TRIPLE \
      -DCMAKE_BUILD_TYPE=Debug
 
-Note how the above cmake command differs from the one used in the other two
-recipes:
+Note how the above cmake command differs from the one used in the other recipe:
 
 * ``clang`` is listed in ``-DLLVM_ENABLE_PROJECTS`` and ``libc`` is
   listed in ``-DLLVM_ENABLE_RUNTIMES``.
@@ -214,7 +129,7 @@ recipes:
 Build step
 ----------
 
-The build step is similar to the other two recipes:
+The build step is similar to the other recipe:
 
 .. code-block:: sh
 

libc/docs/dev/mechanics_of_public_api.rst

@@ -8,27 +8,97 @@ Full Host Build
    :depth: 1
    :local:
 
+.. note:: 
+   Fullbuild requires running headergen, which is a python program that depends on
+   pyyaml. The minimum versions are listed on the :ref:`header_generation`
+   page, as well as additional information.
+
 In this document, we will present a recipe to build the full libc for the host.
 When we say *build the libc for the host*, the goal is to build the libc for
-the same system on which the libc is being built. Also, we will take this
-opportunity to demonstrate how one can set up a *sysroot* (see the documentation
+the same system on which the libc is being built. First, we will explain how to
+build for developing LLVM-libc, then we will explain how to build LLVM-libc as
+part of a complete toolchain.
+
+Configure the build for development
+===================================
+
+
+Below is the list of commands for a simple recipe to build LLVM-libc for
+development. In this we've set the Ninja generator, set the build type to
+"Debug", and enabled the Scudo allocator. This build also enables generating the
+documentation and verbose cmake logging, which are useful development features.
+
+.. note::
+   if your build fails with an error saying the compiler can't find
+   ``<asm/unistd.h>`` or similar then you're probably missing the symlink from
+   ``/usr/include/asm`` to ``/usr/include/<HOST TRIPLE>/asm``. Installing the
+   ``gcc-multilib`` package creates this symlink, or you can do it manually with
+   this command:
+   ``sudo ln -s /usr/include/<HOST TRIPLE>/asm /usr/include/asm``
+   (your host triple will probably be similar to ``x86_64-linux-gnu``)
+
+.. code-block:: sh
+
+   $> cd llvm-project  # The llvm-project checkout
+   $> mkdir build
+   $> cd build
+   $> cmake ../runtimes \
+      -G Ninja \
+      -DCMAKE_C_COMPILER=clang \
+      -DCMAKE_CXX_COMPILER=clang++ \
+      -DLLVM_ENABLE_RUNTIMES="libc;compiler-rt" \
+      -DLLVM_LIBC_FULL_BUILD=ON \
+      -DCMAKE_BUILD_TYPE=Debug \
+      -DLLVM_LIBC_INCLUDE_SCUDO=ON \
+      -DCOMPILER_RT_BUILD_SCUDO_STANDALONE_WITH_LLVM_LIBC=ON \
+      -DCOMPILER_RT_BUILD_GWP_ASAN=OFF                       \
+      -DCOMPILER_RT_SCUDO_STANDALONE_BUILD_SHARED=OFF        \
+      -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
+      -DLLVM_ENABLE_SPHINX=ON -DLIBC_INCLUDE_DOCS=ON \
+      -DLIBC_CMAKE_VERBOSE_LOGGING=ON
+
+Build and test
+==============
+
+After configuring the build with the above ``cmake`` command, one can build test
+libc with the following command:
+
+.. code-block:: sh
+
+   $> ninja libc libm check-libc
+
+To build the docs run this command:
+
+
+.. code-block:: sh
+
+   $> ninja docs-libc-html
+
+To run a specific test, use the following:
+
+.. code-block:: sh
+
+   $> ninja libc.test.src.<HEADER>.<FUNCTION>_test.__unit__
+   $> ninja libc.test.src.ctype.isalpha_test.__unit__ # EXAMPLE
+
+Configure the complete toolchain build
+======================================
+
+For a complete toolchain we recommend creating a *sysroot* (see the documentation
 of the ``--sysroot`` option here:
 `<https://gcc.gnu.org/onlinedocs/gcc/Directory-Options.html>`_) which includes
 not only the components of LLVM's libc, but also a full LLVM only toolchain
 consisting of the `clang <https://clang.llvm.org/>`_ compiler, the
 `lld <https://lld.llvm.org/>`_ linker and the
-`compiler-rt <https://compiler-rt.llvm.org/>`_ runtime libraries. LLVM's libc is
-not yet complete enough to allow using and linking a C++ application against
+`compiler-rt <https://compiler-rt.llvm.org/>`_ runtime libraries. LLVM-libc is
+not quite complete enough to allow using and linking a C++ application against
 a C++ standard library (like libc++). Hence, we do not include
 `libc++ <https://libcxx.llvm.org/>`_ in the sysroot.
 
 .. note:: When the libc is complete enough, we should be able to include
    `libc++ <https://libcxx.llvm.org/>`_, libcxx-abi and libunwind in the
    LLVM only toolchain and use them to build and link C++ applications.
 
-Configure the full libc build
-===============================
-
 Below is the list of commands for a simple recipe to build and install the
 libc components along with other components of an LLVM only toolchain.  In this
 we've set the Ninja generator, enabled a full compiler suite, set the build
@@ -43,6 +113,7 @@ to use the freshly built lld and compiler-rt.
    this command:
    ``sudo ln -s /usr/include/<TARGET TRIPLE>/asm /usr/include/asm``
 
+.. TODO: Move from projects to runtimes for libc, compiler-rt
 .. code-block:: sh
 
    $> cd llvm-project  # The llvm-project checkout
@@ -51,7 +122,7 @@ to use the freshly built lld and compiler-rt.
    $> SYSROOT=/path/to/sysroot # Remember to set this!
    $> cmake ../llvm  \
       -G Ninja  \
-      -DLLVM_ENABLE_PROJECTS="clang;libc;lld;compiler-rt"   \
+      -DLLVM_ENABLE_PROJECTS="clang;lld;libc;compiler-rt"   \
       -DCMAKE_BUILD_TYPE=Debug  \
       -DCMAKE_C_COMPILER=clang \
       -DCMAKE_CXX_COMPILER=clang++ \

libc/docs/dev/source_tree_layout.rst

@@ -4,6 +4,11 @@
 Fullbuild Mode
 ==============
 
+.. note:: 
+   Fullbuild requires running headergen, which is a python program that depends on
+   pyyaml. The minimum versions are listed on the :ref:`header_generation`
+   page, as well as additional information.
+
 The *fullbuild* mode of LLVM's libc is the mode in which it is to be used as
 the only libc (as opposed to the :ref:`overlay_mode` in which it is used along
 with the system libc.) In order to use it as the only libc, one will have to

libc/docs/full_cross_build.rst

@@ -63,9 +63,13 @@ targeting the default host environment as well.
 Runtimes cross build
 --------------------
 
+.. note::
+  These instructions need to be updated for new headergen. They may be
+  inaccurate.
+
 For users wanting more direct control over the build process, the build steps
 can be done manually instead. This build closely follows the instructions in the
-:ref:`main documentation<runtimes_cross_build>` but is specialized for the GPU
+:ref:`main documentation<full_cross_build>` but is specialized for the GPU
 build. We follow the same steps to first build the libc tools and a suitable
 compiler. These tools must all be up-to-date with the libc source.
 

libc/docs/full_host_build.rst

@@ -2,14 +2,16 @@
 The LLVM C Library
 ==================
 
-.. warning::
-  The libc is not complete.  If you need a fully functioning C library right
-  now, you should continue to use your standard system libraries.
+.. note::
+  LLVM-libc is not fully complete right now. Some programs may fail to build due
+  to missing functions (especially C++ ones). If you would like to help us
+  finish LLVM-libc, check out "Contributing to the libc project" in the sidebar
+  or ask on discord.
 
 Introduction
 ============
 
-The libc aspires to a unique place in the software ecosystem.  The goals are:
+LLVM-libc aspires to a unique place in the software ecosystem.  The goals are:
 
 - Fully compliant with current C standards (C17 and upcoming C2x) and POSIX.
 - Easily decomposed and embedded: Supplement or replace system C library
@@ -32,16 +34,17 @@ The libc aspires to a unique place in the software ecosystem.  The goals are:
 Platform Support
 ================
 
-Most development is currently targeting x86_64 and aarch64 on Linux.  Several
-functions in the libc have been tested on Windows.  The Fuchsia platform is
+Most development is currently targeting Linux on x86_64, aarch64, arm, and
+RISC-V. Embedded/baremetal targets are supported on arm and RISC-V, and Windows
+and MacOS have limited support (may be broken).  The Fuchsia platform is
 slowly replacing functions from its bundled libc with functions from this
 project.
 
 ABI Compatibility
 =================
 
 The libc is written to be ABI independent.  Interfaces are generated using
-LLVM's tablegen, so supporting arbitrary ABIs is possible.  In it's initial
+headergen, so supporting arbitrary ABIs is possible.  In it's initial
 stages there is no ABI stability in any form.
 
 .. toctree::

libc/docs/fullbuild_mode.rst

@@ -28,18 +28,18 @@ Also, if users choose to mix more than one libc with the system libc, then
 the name ``libllvmlibc.a`` makes it absolutely clear that it is the static
 archive of LLVM's libc.
 
-Building the static archive with libc as a normal LLVM project
---------------------------------------------------------------
+Building LLVM-libc as a standalone runtime
+------------------------------------------
 
-We can treat the ``libc`` project as any other normal LLVM project and perform
-the CMake configure step as follows:
+We can treat the ``libc`` project like any other normal LLVM runtime library by
+building it with the following cmake command:
 
 .. code-block:: sh
 
   $> cd llvm-project  # The llvm-project checkout
   $> mkdir build
   $> cd build
-  $> cmake ../llvm -G Ninja -DLLVM_ENABLE_RUNTIMES="libc"  \
+  $> cmake ../runtimes -G Ninja -DLLVM_ENABLE_RUNTIMES="libc"  \
      -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ \
      -DCMAKE_BUILD_TYPE=<Debug|Release>                    \  # Select build type
      -DCMAKE_INSTALL_PREFIX=<Your prefix of choice>           # Optional
@@ -50,24 +50,29 @@ Next, build the libc:
 
   $> ninja libc
 
+Then, run the tests:
+
+.. code-block:: sh
+
+  $> ninja check-libc
+
 The build step will build the static archive the in the directory
 ``build/projects/libc/lib``. Notice that the above CMake configure step also
-specified an install prefix. This is optional, but if one uses it, then they
-can follow up the build step with an install step:
+specified an install prefix. This is optional, but it's used, then the following
+command will install the static archive to the install path:
 
 .. code-block:: sh
 
-  $> ninja install-llvmlibc
+  $> ninja install-libc
 
 Building the static archive as part of the bootstrap build
 ----------------------------------------------------------
 
 The bootstrap build is a build mode in which runtime components like libc++,
 libcxx-abi, libc etc. are built using the ToT clang. The idea is that this build
-produces an in-sync toolchain of compiler + runtime libraries. Such a synchrony
-is not essential for the libc but can one still build the overlay static archive
-as part of the bootstrap build if one wants to. The first step is to configure
-appropriately:
+produces an in-sync toolchain of compiler + runtime libraries. This ensures that
+LLVM-libc has access to the latest clang features, which should provide the best
+performance possible.
 
 .. code-block:: sh
 
@@ -77,14 +82,13 @@ appropriately:
      -DCMAKE_BUILD_TYPE=<Debug|Release>                    \  # Select build type
      -DCMAKE_INSTALL_PREFIX=<Your prefix of choice>           # Optional
 
-The build and install steps are similar to the those used when configured
-as a normal project. Note that the build step takes much longer this time
-as ``clang`` will be built before building ``libllvmlibc.a``.
+The build and install steps are the same as above, but the build step will take
+much longer since ``clang`` will be built before building ``libllvmlibc.a``.
 
 .. code-block:: sh
 
   $> ninja libc
-  $> ninja install-llvmlibc
+  $> ninja check-libc
 
 Using the overlay static archive
 ================================

libc/docs/gpu/building.rst

@@ -43,21 +43,6 @@ have their own config directory.
    config directory for Fuchsia as the bring up is being done in the Fuchsia
    source tree.
 
-The api.td file
----------------
-
-If the :ref:`fullbuild_mode` is to be supported on the new operating system,
-then a file named ``api.td`` should be added in its config directory. It is
-written in the
-`LLVM tablegen language <https://llvm.org/docs/TableGen/ProgRef.html>`_.
-It lists all the relevant macros and type definitions we want in the
-public libc header files. See the existing Linux
-`api.td <https://github.com/llvm/llvm-project/blob/main/libc/config/linux/api.td>`_
-file as an example to prepare the ``api.td`` file for the new operating system.
-
-.. note:: In future, LLVM tablegen will be replaced with a different DSL to list
-   config information.
-
 Architecture Subdirectory
 =========================