Skip to content

Commit b82bd2d

Browse files
justinchubygramalingam
justinchuby
and
gramalingam
authoredSep 13, 2023
Merge the two contributing docs and create instructions for updating an op (onnx#5584)
### Description Merge the two contributing docs and create instructions for updating an op ### Motivation and Context Fixes onnx#5379 --------- Signed-off-by: Justin Chu <[email protected]> Co-authored-by: G. Ramalingam <[email protected]>
1 parent 55ffe4e commit b82bd2d

File tree

5 files changed

+203
-179
lines changed

5 files changed

+203
-179
lines changed
 

‎.github/workflows/lint.yml

+2-2
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ jobs:
8686
if ! lintrunner --force-color --all-files --tee-json=lint.json -v; then
8787
echo ""
8888
echo -e "\e[1m\e[36mYou can reproduce these results locally by using \`lintrunner\`.\e[0m"
89-
echo -e "\e[1m\e[36mSee https://github.com/onnx/onnx/blob/main/docs/CONTRIBUTING.md#code-style for setup instructions.\e[0m"
89+
echo -e "\e[1m\e[36mSee https://github.com/onnx/onnx/blob/main/CONTRIBUTING.md#coding-style for setup instructions.\e[0m"
9090
exit 1
9191
fi
9292
- name: Produce SARIF
@@ -107,7 +107,7 @@ jobs:
107107
- name: Check auto-gen files are up-to-date
108108
run: |
109109
echo -e "\n::group:: ===> check auto-gen files are up-to-date..."
110-
110+
111111
ONNX_ML=1 python onnx/defs/gen_doc.py
112112
python onnx/gen_proto.py -l
113113
python onnx/gen_proto.py -l --ml

‎CONTRIBUTING.md

+164-12
Original file line numberDiff line numberDiff line change
@@ -8,29 +8,164 @@ Copyright (c) ONNX Project Contributors
88

99
ONNX is a community project and we welcome your contributions! In addition to contributing code, you can also contribute in many other ways:
1010

11-
* Meetings and Discussions
12-
Join SIGS, Working Groups, Community meetings to learn about what is needed and then where there is a good fit to interest and areas of expertise, find ways to actively contribute. Participate in [ONNX technical discussions](https://github.com/onnx/onnx/discussions) on GitHub. Join the ONNX Slack channels at LF AI and Data, help answer questions and welcome new members.
11+
- Meetings and Discussions
1312

14-
* Use Cases and Tools
15-
Develop use cases for ONNX and advocate for ONNX in developer conferences and meetups. Develop tools that import and export using the ONNX spec, and help grow the community of ONNX users. Become a champion for ONNX in your company or organization.
13+
Join SIGS, Working Groups, Community meetings to learn about what is needed and then where there is a good fit to interest and areas of expertise, find ways to actively contribute. Participate in [ONNX technical discussions](https://github.com/onnx/onnx/discussions) on GitHub. Join the ONNX Slack channels at LF AI and Data, help answer questions and welcome new members.
1614

17-
* Roadmap and Features
18-
Understand the ONNX roadmap document, feature priorities, and help implement them. Become an ONNX code and documentation contributor, and work towards committer status on important repos.
15+
- Use Cases and Tools
1916

20-
* Releases and Model Zoo
21-
Help in achieving a release of ONNX, including increasing the number of models in the ONNX Model Zoo that exercise ONNX features.
17+
Develop use cases for ONNX and advocate for ONNX in developer conferences and meetups. Develop tools that import and export using the ONNX spec, and help grow the community of ONNX users. Become a champion for ONNX in your company or organization.
2218

23-
* Publications and Blogs
24-
Add to the growing number of arXiv papers that refer to ONNX. Create blogs, presentations, books, articles and other materials that help increase the adoption of ONNX, and grow the community of users and contributors.
19+
- Roadmap and Features
2520

26-
* Steering Committee
27-
Attend ONNX Steering Committee meetings - they are open to all in the community. Help out where needed and appropriate on SC to-do items. Note that SIG and Working Groups leaders as well as others with demonstrated commitment and contributions to ONNX community may want to self-nominate during the annual SC election cycle.
21+
Understand the ONNX roadmap document, feature priorities, and help implement them. Become an ONNX code and documentation contributor, and work towards committer status on important repos.
22+
23+
- Releases and Model Zoo
24+
25+
Help in achieving a release of ONNX, including increasing the number of models in the ONNX Model Zoo that exercise ONNX features.
26+
27+
- Publications and Blogs
28+
29+
Add to the growing number of arXiv papers that refer to ONNX. Create blogs, presentations, books, articles and other materials that help increase the adoption of ONNX, and grow the community of users and contributors.
30+
31+
- Steering Committee
32+
33+
Attend ONNX Steering Committee meetings - they are open to all in the community. Help out where needed and appropriate on SC to-do items. Note that SIG and Working Groups leaders as well as others with demonstrated commitment and contributions to ONNX community may want to self-nominate during the annual SC election cycle.
34+
35+
## Adding a new operator or creating a new version of an existing operator
36+
37+
ONNX is an open standard, and we encourage developers to contribute high
38+
quality operators to ONNX specification.
39+
40+
Before proposing a new operator, please read [the tutorial](docs/AddNewOp.md).
2841

2942
## Contributing code
3043

3144
You can submit a pull request (PR) with your code. The [SIG](community/sigs.md) or [Working Group](community/working-groups.md) that is responsible for the area of the project your PR touches will review it and merge once any comments are addressed.
3245

46+
### Development
47+
48+
To build ONNX from source please follow the instructions listed [here](https://github.com/onnx/onnx#build-onnx-from-source).
49+
50+
Then, after you have made changes to Python and C++ files:
51+
52+
- `Python files`: The changes are effective immediately in your installation. You don't need to install these again.
53+
- `C++ files`: You need to install these again to trigger the native extension build.
54+
55+
Assuming build succeed in the initial step, simply running
56+
57+
```sh
58+
pip install -e .
59+
```
60+
61+
from onnx root dir should work.
62+
63+
### Folder structure
64+
65+
- `onnx/`: the main folder that all code lies under
66+
- `onnx.proto`: the protobuf that contains all the structures
67+
- `checker.py`: a utility to check whether a serialized ONNX proto is legal
68+
- `shape_inference.py`: a utility to infer types and shapes for ONNX models
69+
- `version_converter.py`: a utility to upgrade or downgrade version for ONNX models
70+
- `parser.py`: a utility to create an ONNX model or graph from a textual representation
71+
- `hub.py`: a utility for downloading models from [ONNX Model Zoo](https://github.com/onnx/models)
72+
- `compose.py`: a utility to merge ONNX models
73+
- `helper.py`: tools for graph operation
74+
- `defs/`: a subfolder that defines the ONNX operators
75+
- `test/`: test files
76+
77+
### Generated operator documentation
78+
79+
Operator docs ([Operators.md](Operators.md), [Operators-ml.md](Operators-ml.md)) and Changelog docs ([Changelog.md](Changelog.md), [Changelog-ml.md](Changelog-ml.md)) are automatically generated based on C++ operator definitions and backend Python snippets. To refresh all these docs, run the following commands from the repo root and commit the results by setting "ONNX_ML=1". By contrast, setting `ONNX_ML=0` will only update `Operators.md` and `Changelog.md`.
80+
81+
```pwsh
82+
# Windows
83+
set ONNX_ML=1
84+
```
85+
86+
```sh
87+
# UNIX
88+
export ONNX_ML=1
89+
pip install -e .
90+
python onnx/defs/gen_doc.py
91+
```
92+
93+
### Coding style
94+
95+
We use `lintrunner` to drive multiple linters defined in `.lintrunner.toml` to lint the codebase.
96+
97+
To run these checks locally, install `lintrunner` and the linters with
98+
99+
```sh
100+
pip install lintrunner lintrunner-adapters
101+
lintrunner init
102+
```
103+
104+
Then lint with
105+
106+
```sh
107+
lintrunner
108+
```
109+
110+
format with
111+
112+
```sh
113+
# Display all lints and apply the fixes
114+
lintrunner -a
115+
# Or apply fixes only (faster)
116+
lintrunner f
117+
```
118+
119+
Run `lintrunner --help` and see the `.lintrunner.toml` file for more usage examples, as well as instructions on how to adopt new linters.
120+
121+
### Testing
122+
123+
ONNX uses [pytest](https://docs.pytest.org) as a test driver. To run tests, you'll first need to install pytest:
124+
125+
```sh
126+
pip install pytest nbval
127+
```
128+
129+
After installing pytest, run from the root of the repo:
130+
131+
```sh
132+
pytest
133+
```
134+
135+
to run the tests.
136+
137+
<!-- TODO(justinchuby): Get rid of the need for manually running stat_coverage -->
138+
139+
You'll need to regenerate test coverage too, by running this command from the root of the repo:
140+
141+
```sh
142+
python onnx/backend/test/stat_coverage.py
143+
```
144+
145+
#### Cpp tests (googletest)
146+
147+
Some functionalities are tested with googletest. Those tests are listed in `test/cpp`, and include tests for shape inference, data propagation, parser, and others.
148+
149+
To run them, first build ONNX with `-DONNX_BUILD_TESTS=1` or `ONNX_BUILD_TESTS=1 pip install -e .`.
150+
151+
##### Linux and MacOS
152+
153+
The cpp tests require dynamically linking to built libraries.
154+
155+
```sh
156+
export LD_LIBRARY_PATH="./.setuptools-cmake-build/:$LD_LIBRARY_PATH"
157+
.setuptools-cmake-build/onnx_gtests
158+
```
159+
160+
##### Windows
161+
162+
```pwsh
163+
# If you set DEBUG=1, use `.setuptools-cmake-build\Debug\onnx_gtests.exe` instead
164+
.setuptools-cmake-build\Release\onnx_gtests.exe
165+
```
166+
33167
### DCO
168+
34169
ONNX has adopted the [DCO](https://en.wikipedia.org/wiki/Developer_Certificate_of_Origin). All code repositories under ONNX require a DCO. (ONNX previously used a CLA, which is being replaced with the DCO.)
35170

36171
DCO is provided by including a sign-off-by line in commit messages. Using the `-s` flag for `git commit` will automatically append this line. For example, running `git commit -s -m 'commit info.'` it will produce a commit that has the message `commit info. Signed-off-by: My Name <my_email@my_company.com>`. The DCO bot will ensure commits are signed with an email address that matches the commit author before they are eligible to be merged.
@@ -50,3 +185,20 @@ git checkout -b original_patch # create a new branch with the same
50185
git commit -m 'type your own commit msg' -s # signoff that single commit
51186
git push origin original_patch -f # forcibly override the old branch`
52187
```
188+
189+
## CI Pipelines
190+
191+
Every PR needs to pass CIs before merge. CI pipelines details are [here](docs/CIPipelines.md).
192+
193+
## Other developer documentation
194+
195+
- [How to implement ONNX backend (ONNX to something converter)](docs/ImplementingAnOnnxBackend.md)
196+
- [Backend test infrastructure and how to add tests](docs/OnnxBackendTest.md)
197+
198+
## License
199+
200+
[Apache License v2.0](/LICENSE)
201+
202+
## Code of Conduct
203+
204+
[ONNX Open Source Code of Conduct](http://onnx.ai/codeofconduct.html)

‎README.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ ONNX is [widely supported](http://onnx.ai/supported-tools) and can be found in m
4545

4646
ONNX is a community project and the open governance model is described [here](community/readme.md). We encourage you to join the effort and contribute feedback, ideas, and code. You can participate in the [Special Interest Groups](community/sigs.md) and [Working Groups](community/working-groups.md) to shape the future of ONNX.
4747

48-
Check out our [contribution guide](docs/CONTRIBUTING.md) to get started.
48+
Check out our [contribution guide](/CONTRIBUTING.md) to get started.
4949

5050
If you think some operator should be added to ONNX specification, please read
5151
[this document](docs/AddNewOp.md).
@@ -321,7 +321,7 @@ pytest
321321

322322
# Development
323323

324-
Check out the [contributor guide](docs/CONTRIBUTING.md) for instructions.
324+
Check out the [contributor guide](/CONTRIBUTING.md) for instructions.
325325

326326
# License
327327

‎docs/AddNewOp.md

+35-21
Original file line numberDiff line numberDiff line change
@@ -6,37 +6,41 @@ SPDX-License-Identifier: Apache-2.0
66

77
# Adding New Operator or Function to ONNX
88

9+
Or updating an existing operator to a new Opset version.
10+
911
## Table of Contents
1012

1113
- [Adding New Operator or Function to ONNX](#adding-new-operator-or-function-to-onnx)
1214
- [Table of Contents](#table-of-contents)
13-
- [Proposing and submitting a new operator or function to ONNX ](#proposing-and-submitting-a-new-operator-or-function-to-onnx-)
14-
- [4 steps to add an operator ](#4-steps-to-add-an-operator-)
15-
- [Step 1: Proposing a new operator/function ](#step-1-proposing-a-new-operatorfunction-)
16-
- [Step 2: Submit PR ](#step-2-submit-pr-)
15+
- [Proposing and submitting a new operator or function to ONNX](#proposing-and-submitting-a-new-operator-or-function-to-onnx)
16+
- [4 steps to add an operator](#4-steps-to-add-an-operator)
17+
- [Step 1: Proposing a new operator/function](#step-1-proposing-a-new-operatorfunction)
18+
- [Step 2: Submit PR](#step-2-submit-pr)
1719
- [Example to Follow](#example-to-follow)
18-
- [Step 3: PR Review by Operators SIG ](#step-3-pr-review-by-operators-sig-)
20+
- [Step 3: PR Review by Operators SIG](#step-3-pr-review-by-operators-sig)
1921
- [Sign-off](#sign-off)
20-
- [Step 4: ONNX release ](#step-4-onnx-release-)
21-
- [Removing operator or function ](#removing-operator-or-function-)
22-
- [Removing operator ](#removing-operator-)
23-
- [Removing function ](#removing-function-)
24-
- [Document removing operator or function ](#document-removing-operator-or-function-)
22+
- [Step 4: ONNX release](#step-4-onnx-release)
23+
- [Updating an existing operator](#updating-an-existing-operator)
24+
- [Checklist](#checklist)
25+
- [Removing operator or function](#removing-operator-or-function)
26+
- [Removing operator](#removing-operator)
27+
- [Removing function](#removing-function)
28+
- [Document removing operator or function](#document-removing-operator-or-function)
2529

26-
## Proposing and submitting a new operator or function to ONNX <a name="new_operator_or_function"></a>
30+
## Proposing and submitting a new operator or function to ONNX
2731

2832
Operators are the basic building blocks used to define ONNX models. With a rich set of operators, ONNX can describe most DNN and ML models from various frameworks. Functions enable expressing complex operators in terms of more primitive operators. The ONNX specification includes a core set of operators that enable many models. It is a non-goal to add all possible operators, however more operators are added as needed to cover evolving needs.
2933

3034
In this document, we describe the process of accepting a new proposed operator and how to properly submit a new operator as part of ONNX standard. The goal is to improve on what we currently have based on our experience, learning and feedbacks we gathered from the community.
3135

32-
## 4 steps to add an operator <a name="steps_to_add_an_operator"></a>
36+
## 4 steps to add an operator
3337

3438
1. Decide what to propose
3539
2. Submit PR for new operator/function
3640
3. Review of PR by Operators SIG
3741
4. Merging of PR and inclusion in next ONNX release
3842

39-
### Step 1: Proposing a new operator/function <a name="step1_new_operator_or_function"></a>
43+
### Step 1: Proposing a new operator/function
4044

4145
In order to propose a new operator/function, the following is needed:
4246

@@ -54,7 +58,7 @@ This requires carefully balancing generality and complexity. For example, genera
5458
N-D tensors is straight-forward (implementation-wise) for some operators, but complex for other operators.
5559
The choice in such cases will be made based on the complexity of such a generalization.
5660

57-
### Step 2: Submit PR <a name="step2_new_operator_or_function"></a>
61+
### Step 2: Submit PR
5862

5963
Once the criteria of proposing new operator/function has been satisfied, you will need to submit a PR for the new operator/function. Here the expectation of what the PR should include. The reviewer is expected to verify the completeness of the PR before signoff.
6064

@@ -85,21 +89,31 @@ rank inference at the very least (adding right amount of dimensions to the outpu
8589

8690
[PR 1959](https://github.com/onnx/onnx/pull/1959) is a good example to follow.
8791

88-
### Step 3: PR Review by Operators SIG <a name="step3_new_operator_or_function"></a>
92+
### Step 3: PR Review by Operators SIG
8993

9094
The [Operators SIG](https://github.com/onnx/sigs/tree/main/operators) is responsible for the operators/functions in the ONNX specification. The SIG regularly meets and reviews PRs.
9195

9296
#### Sign-off
9397

9498
At least two sign-off from the Operators SIG [contributors](https://github.com/onnx/onnx/tree/main/community#community-roles).
9599

96-
### Step 4: ONNX release <a name="step4_new_operator_or_function"></a>
100+
### Step 4: ONNX release
101+
97102
Once the PR is reviewed and signed off by the Operators SIG, it will be merged. Your new operator/function will be part of the main branch and available to anyone building from source. These are not official releases. ONNX periodically releases official new versions that are a snapshot of the main branch. Your new operator/function will be part of that release.
98103

99-
## Removing operator or function <a name="removing_operator_or_function"></a>
104+
## Updating an existing operator
105+
106+
The definition of an existing operator may need to be updated when e.g. there are new scenarios or input types to support. The process is largely similar to that for creating a new operator.
107+
108+
### Checklist
109+
110+
Use this checklist when updating an existing operator: https://github.com/onnx/onnx/wiki/Checklist-for-updating-an-existing-operator
111+
112+
## Removing operator or function
113+
100114
There are a lot of reasons for removing existing ONNX operator or function, such us being replaced with different operator or can be decomposed by a set of other operators. This document describes the criteria of removing an existing ONNX operator from the standard.
101115

102-
### Removing operator <a name="removing_operator"></a>
116+
### Removing operator
103117

104118
Any operator in ONNX was added because it was required by a model and/or framework. In order to deprecate such an operator we need to do the following.
105119

@@ -111,11 +125,11 @@ Any operator in ONNX was added because it was required by a model and/or framewo
111125
- Add a version adapter which turns the operator into its replacement for the version converter. Example: [onnx/version_converter/adapters/upsample_9_10.h](/onnx/version_converter/adapters/upsample_9_10.h)
112126
- No grace period is needed for deprecated operators.
113127

114-
### Removing function <a name="removing_function"></a>
128+
### Removing function
115129

116-
Function, by definition, is composed of ONNX primitives; however, function could have been accelerated by framework or runtime that support ONNX. So, removing function is not recommended, with the exception of adding another single function which supersede its functionality.
130+
Function, by definition, is composed of ONNX primitives; however, function could have been accelerated by framework or runtime that support ONNX. So, removing function is not recommended, with the exception of adding another single function which supersedes its functionality.
117131

118-
### Document removing operator or function <a name="document_removing_operator_or_function"></a>
132+
### Document removing operator or function
119133

120134
To make sure everyone is aware of the deprecation, the following need to happen:
121135

‎docs/CONTRIBUTING.md

-142
This file was deleted.

0 commit comments

Comments
 (0)
Please sign in to comment.