Header Generation Benchmark in CI (#251)

* Add valgrind to Github env * Add valgrind to Github env * Code format * Disable other benchs for testing * Change action-benchmark version * Fix github workflow * Add comment alert to benchmark CI * IAI output with absolute path * Temporary Logs on Bench CI * More logs in CI * More logs in CI * More logs in CI * More logs in CI * Remove Charts on Github Pages * Git Workflow: update cache to v3 * Add benchmark to CI * Benchs run after build in reference machine * CI Benchs: fix cargo path * CI Benchs: Bench in parallel with build * CI Benchs: Bench sscache fix * CI Benchs: Missing deps * CI Benchs: log caches * CI Benchs: cache more stuff * CI Benchs: add Criterion * CI Benchs: Increase threashold * CI Benchs: bencher output format * CI Benchs: fix some paths * CI Benchs: abs path for iai outputs * CI Benchs: more logs to benchs * CI Benchs: more logs to benchs * CI Benchs: reuse a workflow * CI Benchs: no-reusable workflow * CI Benchs: syntax fix * CI Benchs: Remove unused file * CI Benchs: Remove unused file * CI Benchs: Remove codspeed from ref machine * CI Benchs: Remove codspeed from ref machine * CI Benchs: Remove codspeed from ref machine * CI Benchs: more logs to benchs * CI Benchs: more logs to benchs * CI Benchs: more logs to benchs * CI Benchs: fix benches exit code * CI Benchs: create target folders * CI Benchs: E2E improvements * CI Benchs: SCCache bin is cached * CI Benchs: UTs need WASM generation * CI Benchs: Integrate CodSpeed * CI Benchs: Integrate CodSpeed * CI Benchs: Remove CodSpeed * CI Benchs: Remove CodSpeed * Restore `run-benchmarks` action * Added Divan and iai callgrind benchmarks. Moved benchmarks into a separate workflow * Removed code coverage report * Commit just to trigger CI * Moved unit tests out of default workflow --------- Co-authored-by: Marko Petrlic <[email protected]>
availproject · Nov 4, 2023 · 33f7951 · 33f7951 · github-actions · Nov 4, 2023
1 parent 5e12b25
commit 33f7951
Show file tree

Hide file tree

Showing 15 changed files with 759 additions and 142 deletions.
diff --git a/.github/workflows/benchmarks.yml b/.github/workflows/benchmarks.yml
@@ -0,0 +1,178 @@
+name: Header and RPC Benchmarks
+on:
+  push:
+    branches:
+      - main
+      - develop
+  pull_request:
+    branches:
+      - main
+      - develop
+  workflow_dispatch:
+
+env:
+  CARGO_SCCACHE_COMMIT: bed5571c
+
+jobs:
+  benchmarks_iai:
+    runs-on: [self-hosted, reference]
+    env:
+      SKIP_WASM_BUILD: true
+      RUST_BACKTRACE: full
+    steps:
+      - uses: actions/checkout@v2
+
+      - uses: actions/cache@v3
+        with:
+          path: |
+            ~/.cache/
+            ~/.cargo/bin/sccache*
+          key: ${{ runner.OS }}-cache-iai-${{ env.CARGO_SCCACHE_COMMIT }}-v1
+
+      - name: Install system dependencies
+        # We force to reinstall `valgrind` because `codspeed` replaces that binary and the output
+        # of `valgrind.codspeed` generates a 
+
+        run: |
+          sudo apt-get update
+          sudo apt remove -y valgrind
+          sudo apt-get install -y build-essential pkg-config libssl-dev valgrind git clang curl libssl-dev protobuf-compiler unzip python3-pip
+
+      - name: Setup Rust toolchain
+        run: |
+          curl https://sh.rustup.rs -sSf | sh -s -- -y
+          source "$HOME/.cargo/env"
+          rustup show
+
+      - name: Set PATH for cargo
+        run: |
+          echo "$HOME/.cargo/bin" >> $GITHUB_PATH
+          echo "SCCACHE_DIR=${HOME}/.cache/sccache" >> $GITHUB_ENV
+
+      - name: SCCache
+        run: |
+          # We altered the path to avoid old actions to overwrite it
+          if [ ! -f ~/.cargo/bin/sccache ]; then
+            cargo install sccache \
+              --git https://github.com/purestake/sccache.git \
+              --rev $CARGO_SCCACHE_COMMIT \
+              --force --no-default-features --features=dist-client
+          fi
+
+          if [[ -z `pgrep sccache` ]]; then
+            chmod +x ~/.cargo/bin/sccache
+            sccache --start-server
+          fi
+          sccache -s
+          echo "RUSTC_WRAPPER=${HOME}/.cargo/bin/sccache" >> $GITHUB_ENV
+
+      - name: Run Header Generation Benchmarks on IAI
+        run: |
+          cargo uninstall --locked cargo-codspeed | true
+          which valgrind
+          sudo dpkg -S `which valgrind`
+          valgrind --version
+          mkdir -p ~/.cache/iai/
+          cargo bench -p da-runtime --bench header_kate_commitment_iai | tee ~/.cache/iai/header_gen_bench_iai.txt || true
+          find ./runtime/target/iai -type f
+
+      - name: Header Generation Regression Checks on IAI
+        uses: fmiguelgarcia/github-action-benchmark@v1
+        with:
+          # What benchmark tool the output.txt came from
+          tool: 'rustIai'
+          # Where the output from the benchmark tool is stored
+          output-file-path: ~/.cache/iai/header_gen_bench_iai.txt
+          # Where the previous data file is stored
+          external-data-json-path: ~/.cache/iai/benchmark-data-iai.json
+          save-data-file: true
+          # Workflow will fail when an alert happens at 15% degradation
+          fail-on-alert: true
+          alert-threshold: '115%'
+          # Upload the updated cache file for the next job by actions/cache
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          # Enable alert commit comment
+          comment-on-alert: true
+          # Mention @rhysd in the commit comment
+          alert-comment-cc-users: '@prabal-banerjee,@jakubcech,@vthunder,@kroos47,@Leouarz,@markopoloparadox'
+          comment-always: true
+          summary-always: true
+
+  benchmarks_cri:
+    runs-on: [self-hosted, reference]
+    env:
+      SKIP_WASM_BUILD: true
+      RUST_BACKTRACE: full
+    steps:
+      - uses: actions/checkout@v2
+
+      - uses: actions/cache@v3
+        with:
+          path: |
+            ~/.cache/
+            ~/.cargo/bin/sccache*
+          key: ${{ runner.OS }}-cache-cri-${{ env.CARGO_SCCACHE_COMMIT }}-v1
+
+      - name: Install system dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y build-essential pkg-config libssl-dev valgrind git clang curl libssl-dev protobuf-compiler unzip python3-pip
+
+      - name: Setup Rust toolchain
+        run: |
+          curl https://sh.rustup.rs -sSf | sh -s -- -y
+          source "$HOME/.cargo/env"
+          rustup show 
+
+      - name: Set PATH for cargo
+        run: |
+          echo "$HOME/.cargo/bin" >> $GITHUB_PATH
+          echo "SCCACHE_DIR=${HOME}/.cache/sccache" >> $GITHUB_ENV
+
+      - name: SCCache
+        run: |
+          # We altered the path to avoid old actions to overwrite it
+          if [ ! -f ~/.cargo/bin/sccache ]; then
+            cargo install sccache \
+              --git https://github.com/purestake/sccache.git \
+              --rev $CARGO_SCCACHE_COMMIT \
+              --force --no-default-features --features=dist-client
+          fi
+
+          if [[ -z `pgrep sccache` ]]; then
+            chmod +x ~/.cargo/bin/sccache
+            sccache --start-server
+          fi
+          sccache -s
+          echo "RUSTC_WRAPPER=${HOME}/.cargo/bin/sccache" >> $GITHUB_ENV
+
+      - name: Run Header Generation Benchmarks on Criterion 
+        run: |
+          mkdir -p ~/.cache/cri/
+          cargo bench -p da-runtime --bench header_kate_commitment_cri -- --color never --noplot --output-format bencher | tee ~/.cache/cri/header_gen_bench_cri.txt || true
+
+      - name: Header Generation Regression Checks on Criterion 
+        uses: fmiguelgarcia/github-action-benchmark@v1
+        with:
+          # What benchmark tool the output.txt came from
+          tool: 'cargo'
+          # Where the output from the benchmark tool is stored
+          output-file-path: ~/.cache/cri/header_gen_bench_cri.txt
+          # Where the previous data file is stored
+          external-data-json-path: ~/.cache/cri/benchmark-data-cri.json
+          save-data-file: true
+          # Workflow will fail when an alert happens at 15% degradation
+          fail-on-alert: true
+          alert-threshold: '115%'
+          # Upload the updated cache file for the next job by actions/cache
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          # Enable alert commit comment
+          comment-on-alert: true
+          # Mention @rhysd in the commit comment
+          alert-comment-cc-users: '@prabal-banerjee,@jakubcech,@vthunder,@kroos47,@Leouarz,@markopoloparadox'
+          comment-always: true
+          summary-always: true
+
+      - name: Display SCCache Stats
+        run: sccache --show-stats
+
diff --git a/.github/workflows/default.yml b/.github/workflows/default.yml
@@ -1,4 +1,4 @@
-name: Lint and Unit Tests
+name: Lint and Features check
 on:
   push:
     branches:
@@ -12,8 +12,6 @@ on:
 
 env:
   CARGO_SCCACHE_COMMIT: bed5571c
-  SCCACHE_DIR: /home/runner/.cache/cargo-sccache-bed5571c
-  SCCACHE_BIN: /home/runner/.cache/cargo-sccache-bed5571c/bin/sccache
 
 jobs:
   lint:
@@ -56,34 +54,44 @@ jobs:
 
       - uses: actions/cache@v3
         with:
-          path: ${{ env.SCCACHE_DIR }}
+          path: |
+            ~/.cache/
+            ~/.cargo/bin/sccache*
           key: ${{ runner.OS }}-sccache-bin-${{ env.CARGO_SCCACHE_COMMIT }}-v1
 
-      - name: Install Protoc
-        uses: arduino/setup-protoc@v1
-        with:
-          version: "3.x"
+      - name: Install system dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y build-essential pkg-config libssl-dev valgrind git clang curl libssl-dev protobuf-compiler unzip python3-pip
+
 
       - name: Setup Rust toolchain
         run: |
           curl https://sh.rustup.rs -sSf | sh -s -- -y
           source "$HOME/.cargo/env"
           rustup show
 
+      - name: Set PATH for cargo
+        run: |
+          echo "$HOME/.cargo/bin" >> $GITHUB_PATH
+          echo "SCCACHE_DIR=${HOME}/.cache/sccache" >> $GITHUB_ENV
+
       - name: SCCache
         run: |
           # We altered the path to avoid old actions to overwrite it
-          if [ ! -f $SCCACHE_BIN ]; then
-            cargo install sccache --git https://github.com/purestake/sccache.git --rev $CARGO_SCCACHE_COMMIT --force --no-default-features --features=dist-client --root $SCCACHE_DIR
+          if [ ! -f ~/.cargo/bin/sccache ]; then
+            cargo install sccache \
+              --git https://github.com/purestake/sccache.git \
+              --rev $CARGO_SCCACHE_COMMIT \
+              --force --no-default-features --features=dist-client
           fi
-          ls -la $SCCACHE_BIN
-          ps aux | grep sccache
+
           if [[ -z `pgrep sccache` ]]; then
-            chmod +x $SCCACHE_BIN
-            $SCCACHE_BIN --start-server
+            chmod +x ~/.cargo/bin/sccache
+            sccache --start-server
           fi
-          $SCCACHE_BIN -s
-          echo "RUSTC_WRAPPER=$SCCACHE_BIN" >> $GITHUB_ENV
+          sccache -s
+          echo "RUSTC_WRAPPER=${HOME}/.cargo/bin/sccache" >> $GITHUB_ENV
 
       - name: Build node
         run: cargo build --release -p data-avail
@@ -95,71 +103,55 @@ jobs:
           path: target/release/data-avail
 
       - name: Display SCCache Stats
-        run: ${{ env.SCCACHE_BIN }} --show-stats
+        run: sccache --show-stats
 
-      - name: Check other features
-        run: cargo check --release --workspace --features "runtime-benchmarks try-runtime" -p data-avail
-
-  unit_tests:
+  check_features:
     runs-on: ubuntu-latest
     needs: [build]
     steps:
       - uses: actions/checkout@v2
 
-      - name: Free Disk Space (Ubuntu)
-        uses: jlumbroso/free-disk-space@main
-        with:
-          tool-cache: true
-
-      - name: Install build-essential
-        run: |
-          sudo apt update
-          sudo apt install -y build-essential
-          sudo apt install -y git clang curl libssl-dev protobuf-compiler
-
-      # Restore cache from `build`
       - uses: actions/cache/restore@v3
         with:
-          path: ${{ env.SCCACHE_DIR }}
-          key: ${{ runner.OS }}-sccache-bin-${{ env.CARGO_SCCACHE_COMMIT }}-v1
+          path: |
+            ~/.cache/
+            ~/.cargo/bin/sccache*
+          key: ${{ runner.OS }}-cache-bin-${{ env.CARGO_SCCACHE_COMMIT }}-v1
 
-      - name: Install Protoc
-        uses: arduino/setup-protoc@v1
-        with:
-          version: "3.x"
+      - name: Install system dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y build-essential pkg-config libssl-dev valgrind git clang curl libssl-dev protobuf-compiler unzip python3-pip
 
       - name: Setup Rust toolchain
         run: |
           curl https://sh.rustup.rs -sSf | sh -s -- -y
           source "$HOME/.cargo/env"
           rustup show
 
+      - name: Set PATH for cargo
+        run: |
+          echo "$HOME/.cargo/bin" >> $GITHUB_PATH
+          echo "SCCACHE_DIR=${HOME}/.cache/sccache" >> $GITHUB_ENV
+
       - name: SCCache
         run: |
           # We altered the path to avoid old actions to overwrite it
-          if [ ! -f $SCCACHE_BIN ]; then
-            cargo install sccache --git https://github.com/purestake/sccache.git --rev $CARGO_SCCACHE_COMMIT --force --no-default-features --features=dist-client --root $SCCACHE_DIR
+          if [ ! -f ~/.cargo/bin/sccache ]; then
+            cargo install sccache \
+              --git https://github.com/purestake/sccache.git \
+              --rev $CARGO_SCCACHE_COMMIT \
+              --force --no-default-features --features=dist-client
           fi
-          ls -la $SCCACHE_BIN
-          ps aux | grep sccache
+
           if [[ -z `pgrep sccache` ]]; then
-            chmod +x $SCCACHE_BIN
-            $SCCACHE_BIN --start-server
+            chmod +x ~/.cargo/bin/sccache
+            sccache --start-server
           fi
-          $SCCACHE_BIN -s
-          echo "RUSTC_WRAPPER=$SCCACHE_BIN" >> $GITHUB_ENV
+          sccache -s
+          echo "RUSTC_WRAPPER=${HOME}/.cargo/bin/sccache" >> $GITHUB_ENV
 
-      - name: Run tests
-        run: |
-          env
-          cargo test --release --workspace
-          cargo test --release --manifest-path avail-subxt/Cargo.toml
+      - name: Check other features
         env:
-          RUSTFLAGS: "-C instrument-coverage"
-          LLVM_PROFILE_FILE: "profile-%p-%m.profraw"
-
-      - name: Display SCCache Stats
-        run: ${{ env.SCCACHE_BIN }} --show-stats
-
-      - name: Cleanup
-        run: find . -name \*.profraw -type f -exec rm -f {} +
+          SKIP_WASM_BUILD: true
+        run: cargo check --release --workspace --features "runtime-benchmarks try-runtime" -p data-avail
diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml
@@ -103,7 +103,7 @@ jobs:
           with:
             name: data-avail
 
-        - name: Run E2E SubXt tests
+        - name: Run Kate RPC E2E tests
           run: |
             chmod uog+x data-avail
             ./data-avail --dev &
@@ -153,7 +153,7 @@ jobs:
         with:
           name: data-avail
 
-      - name: Run E2E SubXt tests
+      - name: Run SubXt E2E tests
         run: |
           chmod uog+x data-avail
           ./data-avail --dev &

diff --git a/.github/workflows/releaser.yml b/.github/workflows/releaser.yml
@@ -5,6 +5,7 @@ on:
       - '**'
     tags:
       - '*'
+  workflow_dispatch:
 
 jobs:
   binary_linux_amd64:
-Original file line number
+Diff line change
@@ Expand Up / @@ -5,6 +5,7 @@ on: @@
           - '**'
         tags:
           - '*'
+      workflow_dispatch:
     jobs:
       binary_linux_amd64:
@@ Expand Down @@
Benchmark suite	Current: `33f7951`	Previous: `d5e85c3`	Ratio
`commitment_builder/32`	`4325417137` ns/iter (`± 81695432`)	`4359561877` ns/iter (`± 123327154`)	`0.99`
`commitment_builder/64`	`4343451680` ns/iter (`± 92231152`)	`4188904750` ns/iter (`± 129943054`)	`1.04`
`commitment_builder/128`	`4647446215` ns/iter (`± 137494514`)	`4533002317` ns/iter (`± 206935699`)	`1.03`
`commitment_builder/256`	`2367397557` ns/iter (`± 42776582`)	`2302998591` ns/iter (`± 68415605`)	`1.03`
Benchmark suite	Current: `33f7951`	Previous: `d5e85c3`	Ratio
`commitment_builder_32 Instructions`	`43503795372` ops (`± 0`)	`42936286901` ops (`± 0`)	`1.01`
`commitment_builder_32 L1 Accesses`	`53515809463` ops (`± 0`)	`52851148072` ops (`± 0`)	`1.01`
`commitment_builder_32 L2 Accesses`	`735327` ops (`± 0`)	`441028` ops (`± 0`)	`1.67`
`commitment_builder_32 RAM Accesses`	`92091` ops (`± 0`)	`98216` ops (`± 0`)	`0.94`
`commitment_builder_32 Estimated Cycles`	`53522709283` ops (`± 0`)	`52856790772` ops (`± 0`)	`1.01`
`commitment_builder_64 Instructions`	`41972715363` ops (`± 0`)	`41407144622` ops (`± 0`)	`1.01`
`commitment_builder_64 L1 Accesses`	`51407644825` ops (`± 0`)	`50745546732` ops (`± 0`)	`1.01`
`commitment_builder_64 L2 Accesses`	`892440` ops (`± 0`)	`591169` ops (`± 0`)	`1.51`
`commitment_builder_64 RAM Accesses`	`191104` ops (`± 0`)	`199963` ops (`± 0`)	`0.96`
`commitment_builder_64 Estimated Cycles`	`51418795665` ops (`± 0`)	`50755501282` ops (`± 0`)	`1.01`
`commitment_builder_128 Instructions`	`54204380871` ops (`± 0`)	`53637645012` ops (`± 0`)	`1.01`
`commitment_builder_128 L1 Accesses`	`68302239061` ops (`± 0`)	`67638582208` ops (`± 0`)	`1.01`
`commitment_builder_128 L2 Accesses`	`1276967` ops (`± 0`)	`1006109` ops (`± 0`)	`1.27`
`commitment_builder_128 RAM Accesses`	`243077` ops (`± 0`)	`265631` ops (`± 0`)	`0.92`
`commitment_builder_128 Estimated Cycles`	`68317131591` ops (`± 0`)	`67652909838` ops (`± 0`)	`1.01`
`commitment_builder_256 Instructions`	`31758245732` ops (`± 0`)	`31191865472` ops (`± 0`)	`1.02`
`commitment_builder_256 L1 Accesses`	`40832842847` ops (`± 0`)	`40169818877` ops (`± 0`)	`1.02`
`commitment_builder_256 L2 Accesses`	`6776904` ops (`± 0`)	`6494397` ops (`± 0`)	`1.04`
`commitment_builder_256 RAM Accesses`	`237457` ops (`± 0`)	`266583` ops (`± 0`)	`0.89`
`commitment_builder_256 Estimated Cycles`	`40875038362` ops (`± 0`)	`40211621267` ops (`± 0`)	`1.02`