Update to Testgrid Documentation

README.md

@@ -89,7 +89,6 @@ environments. For more info please see [Contributing Test Results](docs/contribu
 [`velodrome`]: /velodrome
 [`testgrid`]: /testgrid
 [testgrid.k8s.io]: https://testgrid.k8s.io
-[`testgrid/config.yaml`]: /testgrid/config.yaml
 [`triage`]: /triage
 [velodrome/bigquery-metrics]: http://velodrome.k8s.io/dashboard/db/bigquery-metrics?orgId=1
 [velodrome/monitoring]: http://velodrome.k8s.io/dashboard/db/monitoring?orgId=1

config/README.md

@@ -1,11 +1,11 @@
 # Kubernetes Project Configs
 
 This is a central place for Kubernetes-project specific configs for some of
-the tools housed within this repo. At present, this is just [prow.k8s.io]
+the tools housed within this repo.
 
 ### Directory Structure
 
-- [`jobs/`](./jobs): prow job configs for [prow.k8s.io]
+- [`jobs/`](./jobs): Prow job configs for [prow.k8s.io](https://prow.k8s.io)
+- [`testgrids/`](./testgrids): Testgrid configuration for [testgrid.k8s.io](https://testgrid.k8s.io)
 - [`tests/`](./tests): validation tests for the configs
 
-[prow.k8s.io]: https://prow.k8s.io

config/jobs/README.md

@@ -142,9 +142,8 @@ periodics:
 1. Ensure an [`OWNERS`](https://go.k8s.io/owners) file exists in the directory for job, and has appropriate approvers/reviewers
 1. Write or edit the job config (please see [how-to-add-new-jobs](/prow/jobs.md#how-to-configure-new-jobs))
 1. Ensure the job is configured to to display its results in [testgrid.k8s.io]
-    - The new way: add [testgrid annotations]
-    - The old way: update [`testgrid/config.yaml`]
-    - Please see the [testgrid README](/testgrid/README.md) for more details on configuation options
+    - The simple way: add [testgrid annotations]
+    - Please see the testgrid [documentation](/testgrid/config.md) for more details on configuation options
 1. Ensure any necessary formatting or config generation happens
     - Run [`hack/update-config.sh`]
     - CI will fail against your PR if this was necessary
@@ -212,13 +211,12 @@ bazel run //experiment/config-forker -- \
 
 [prow.k8s.io]: https://prow.k8s.io
 [@k8s-ci-robot]: https://github.com/k8s-ci-robot
-[testgrid annotations]: /testgrid/config.md#minimal-configuration
+[testgrid annotations]: /testgrid/config.md#prow-job-configuration
 [testgrid.k8s.io]: https://testgrid.k8s.io
 
 [`experiment/config-forker`]: /experiment/config-forker
 [`hack/update-config.sh`]: /hack/update-config.sh
 [`images/`]: /images
-[`testgrid/config.yaml`]: /testgrid/config.yaml
 
 [periodic-kubernetes-e2e-packages-pushed]: https://github.com/kubernetes/test-infra/blob/688d365adf7f71e33a4249c7b90d7e84c105dfc5/config/jobs/kubernetes/sig-cluster-lifecycle/packages.yaml#L3-L16
 [pull-community-verify]: https://github.com/kubernetes/test-infra/blob/f4e6553b27d9ee8b35b2f2e588ea2e18c3fa818b/config/jobs/kubernetes/community/community-presubmit.yaml#L3-L19

config/testgrids/README.md

@@ -1,8 +1,23 @@
 # Testgrid Configurations
 
-See [config.md](../../testgrid/config.md) for how to configure testgrid
+This readme covers information specific to testgrid.k8s.io and this repository.
+See Testgrid's [config.md](../../testgrid/config.md) for more information Testgrid config files.
 
-For these files, do not set your own defaults; they're still global variables and may affect
-other configurations.
+## Adding a Prow Job to Testgrid
 
-TODO(slchase): Further documentation
+Prow Jobs in this repository only need to be [annotated](/testgrid/config.md#prow-job-configuration);
+no changes are necessary here unless you are adding a brand new dashboard.
+
+## Adding or Changing a Configuration
+
+Any file put in this directory or a subdirectory will be picked up by [testgrid.k8s.io](https://testgrid.k8s.io).
+
+## Testing
+
+Run `bazel test //testgrid/...` to ensure the configuration is valid.
+
+This finds common problems such as malformed yaml, a tab referring to a
+non-existent test group, a test group never appearing on any tab, etc.
+
+Run `bazel test //...` for slightly more advanced testing, such as ensuring that
+every job in our CI system appears somewhere in testgrid, etc.

docs/contributing-test-results.md

@@ -17,7 +17,7 @@ The process is as follows:
     rewriting it to better support external usage.
 - Add the GCS bucket info to [buckets.yaml](/kettle/buckets.yaml) via a PR (use the
   previously designated github handle for the `contact` field).
-- Add jobs and dashboards to the [testgrid config](/testgrid/config.yaml) via
+- Add jobs and dashboards to the [testgrid config](/config/testgrids) via
   a PR (use the previously designated point of contact info in a comment next to
   added `test_group`s, or even better in the `description` field for added
   `dashboard_tab`s)

testgrid/README.md

@@ -106,13 +106,10 @@ Under **Options**
 If you need to add a new test that you want TestGrid to display, or otherwise change what is shown
 on https://testgrid.k8s.io, see [Testgrid Configuration](config.md).
 
-The site is configured by [`config.yaml`].
 Updates to the config are automatically tested and pushed to production.
 
 ## Contributing
 
 If you want to modify TestGrid beyond adding new tests or dashboards, see [Updating Testgrid](build_test_update.md).
 
-[`config.proto`]: ./config/config.proto
-[`config.yaml`]: ./config.yaml
 [video]: https://www.youtube.com/watch?v=jm2l2SLq_yE

testgrid/build_test_update.md

@@ -5,15 +5,15 @@ If you need to add a new test that you want TestGrid to display, see [TestGrid C
 If you're looking to develop for TestGrid, welcome! Note that most of the inner workings of TestGrid
 are not open source. This is a [known issue](https://github.com/kubernetes/test-infra/issues/10409).
 
-## config.yaml and config.proto
+## YAML configuration and config.proto
 
 While TestGrid configuration is located in YAML files, TestGrid doesn't natively
 read YAML. Instead, it expects configuration data in [`config.proto`]. This file
 is commented, and should be treated as the authoritative "input" schema to
 TestGrid.
 
 [`config.proto`] is generated primarily with [Configurator](cmd/configurator).
-Updates to [`config.yaml`] are automatically Configurated when a change is
+Updates to the [testgrid.k8s.io config] are automatically Configurated when a change is
 merged.
 
 You can convert a yaml file to the config proto with:
@@ -42,10 +42,10 @@ to verify.
 ## Testing
 
 Run `bazel test //testgrid/...` to run all unit tests in TestGrid. Note that this also validates
-the contents of [`config.yaml`]
+the [testgrid.k8s.io config].
 
-Run `bazel test //...` for slightly more advanced testing, such as ensuring that
+Run `bazel test //...` for repository-wide testing, such as ensuring that
 every job in our CI system appears somewhere in testgrid, etc.
 
 [`config.proto`]: ./config/config.proto
-[`config.yaml`]: ./config.yaml
+[testgrid.k8s.io config]: /config/testgrids

testgrid/cmd/configurator/README.md

@@ -0,0 +1,61 @@
+# Configurator
+
+Configurator takes some YAML Testgrid config files and (optionally) a Prow configuration and generates
+a complete Testgrid configuration. 
+
+This utility is important for the [inner workings](/testgrid/build_test_update.md) of Testgrid, but if
+you're looking to just add to or modify an existing configuration, read [`config.md`]
+instead.
+
+## Basic Usage
+
+`configurator --yaml=config.yaml --print-text --oneshot` will read the configuration from the YAML
+file and print it to standard output for humans to read.
+
+If `--oneshot` is omitted, it will do the same thing and continue running. If the configuration it's
+reading is modified, it will generate again.
+
+Instead of `--print-text`, you can just `--validate-config-file`, or specify an `--output`.
+
+```bash
+--output=/path/outputfile     # Writes the generated configuration to that file
+--output=gcs://bucket/object  # Writes the generated configuration to a GCS bucket. Credentials are needed.
+```
+
+`--default` specifies default settings to use whenever a setting isn't specified in the YAML configuration.
+
+## Usage with Prow
+
+If Testgrid is running in parallel with [Prow], configuration can be annotated to a Prow job instead
+of separately configured in a YAML file. Details for how to write these annotations are in [`config.md`].
+
+The options `--prow-config` and `--prow-job-config` are used to specify where the Prow configurations are.
+They must be specified together.
+
+## Deserialization Options
+
+Configurator reads YAML configurations. Testgrid itself expects its configuration to be formatted as
+a [protocol buffer][`config.proto`], and has no concept of a YAML configuration.
+
+By default, Configurator outputs a [`config.proto`], since it usually serves configurations to Testgrid.
+However, if you want to use Configurator to generate output that will be consumed by a _different
+instance_ of Configurator, you may want to use the `--output-yaml` option.
+
+For example, if you are running your own instance of Prow, want to use annotations,
+and need to output those annotations so that the k8s instance of Configurator can parse them correctly,
+you may want to use a command like the following:
+
+```bash
+configurator \
+    --yaml=./dashboard_list.yaml \
+    --default=./project_defaults.yaml \
+    --prow-config=./prow_config.yaml \
+    --prow-job-config=./prow_jobs \
+    --output=./output_config.yaml \
+    --output-yaml \
+    --oneshot
+```
+
+[`config.proto`]: /testgrid/config/config.proto
+[`config.md`]: /testgrid/config.md
+[Prow]: /prow/README.md

testgrid/config.md

@@ -14,18 +14,21 @@ Testgrid is composed of:
 * A list of **dashboards**, or collections of dashboard tabs
 * A list of **dashboard groups** of related dashboards.
 
-Most of these objects are listed in [`config.yaml`]
+Most of these objects are simply listed in a [YAML config file][configuration] for Testgrid to consume.
 
 ## Prow Job Configuration
 
-If you just have a prowjob you want to appear in an existing dashboard, add annotations to that
-prowjob. You don't need to change a config file here.
+If you just have a [Prow job](/prow/jobs.md) configuration you want to appear in an existing
+dashboard, add annotations to that Prow job.
 
-Simply add this to your prowjob:
+If it's a Prow job in [the k8s.io instance](/config/jobs), you don't need to do anything else.
+
+
+Add this to your Prow job:
 
 ```yaml
 annotations:
-  testgrid-dashboards: dashboard-name      # a dashboard defined in config.yaml.
+  testgrid-dashboards: dashboard-name      # a dashboard already defined in a config.yaml.
   testgrid-tab-name: some-short-name       # optionally, a shorter name for the tab. If omitted, just uses the job name.
   testgrid-alert-email: [email protected]          # optionally, an alert email that will be applied to the tab created in the
                                            # first dashboard specified in testgrid-dashboards.
@@ -39,12 +42,13 @@ annotations:
 
 ```
 
-This functionality is provided by [Configurator](cmd/configurator).
+This functionality is provided by [Configurator](cmd/configurator). If you have Prow jobs in a _different_
+instance of Prow, you may want to invoke Configurator [differently](cmd/configurator#deserialization-options).
 
 If you need to create a new dashboard, or do anything more advanced, read on.
 
 ## Configuration
-Open [`config.yaml`] in your favorite editor and:
+Open or create a Testgrid config file [(example)][configuration] in your favorite editor and:
 1. Configure the test groups
 2. Add those testgroups to one or more tabs in one or more dashboards
 3. Consider using dashboard groups if multiple dashboards are needed.
@@ -113,14 +117,11 @@ dashboard_groups:
 
 ## Testing your configuration
 
-Run `bazel test //testgrid/...` to ensure the config is valid.
+Run `bazel test //testgrid/...` to ensure the configuration is valid.
 
 This finds common problems such as malformed yaml, a tab referring to a
 non-existent test group, a test group never appearing on any tab, etc.
 
-Run `bazel test //...` for slightly more advanced testing, such as ensuring that
-every job in our CI system appears somewhere in testgrid, etc.
-
 ## Advanced configuration
 See [`config.proto`] for an extensive list of configuration options. Here are some commonly-used ones.
 
@@ -325,4 +326,4 @@ test_groups:
 ```
 
 [`config.proto`]: ./config/config.proto
-[`config.yaml`]: ./config.yaml
+[configuration]: /config/testgrids