Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: readium/go-toolkit
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: main
Choose a base ref
...
head repository: readium/go-toolkit
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: develop
Choose a head ref
Can’t automatically merge. Don’t worry, you can still create the pull request.
Loading
Showing with 8,678 additions and 3,279 deletions.
  1. +0 −13 .eslintrc.js
  2. +24 −0 .github/dependabot.yml
  3. +14 −5 .github/workflows/build.yml
  4. +78 −0 .github/workflows/codeql.yml
  5. +27 −0 .github/workflows/dependency-review.yml
  6. +0 −30 .github/workflows/release.yml
  7. +76 −0 .github/workflows/scorecards.yml
  8. +1 −2 .gitignore
  9. +0 −39 .goreleaser.yaml
  10. +14 −0 .pre-commit-config.yaml
  11. +50 −6 CHANGELOG.md
  12. 0 LICENSE.md → LICENSE
  13. +0 −15 Makefile
  14. +2 −55 README.md
  15. +0 −123 cmd/rwp/cmd/manifest.go
  16. +0 −27 cmd/rwp/cmd/root.go
  17. +0 −7 cmd/rwp/main.go
  18. +0 −5 cmd/server/README.MD
  19. +0 −242 cmd/server/api/server.go
  20. +0 −11 cmd/server/api/structs.go
  21. +0 −1 cmd/server/configs/.gitignore
  22. +0 −12 cmd/server/configs/config.local.toml.example
  23. +0 −124 cmd/server/internal/config/config.go
  24. +0 −7 cmd/server/internal/consts/consts.go
  25. +0 −57 cmd/server/internal/logging/logging.go
  26. +0 −46 cmd/server/main.go
  27. +75 −32 go.mod
  28. +216 −395 go.sum
  29. +389 −0 pkg/analyzer/image.go
  30. +32 −10 pkg/archive/archive.go
  31. +25 −4 pkg/archive/archive_exploded.go
  32. +102 −0 pkg/archive/archive_gcs.go
  33. +143 −0 pkg/archive/archive_http.go
  34. +125 −0 pkg/archive/archive_s3.go
  35. +3 −1 pkg/archive/archive_test.go
  36. +127 −9 pkg/archive/archive_zip.go
  37. +10 −0 pkg/archive/compression.go
  38. +12 −0 pkg/archive/gzip.go
  39. +342 −0 pkg/archive/remote.go
  40. +38 −17 pkg/asset/asset_file.go
  41. +144 −0 pkg/asset/asset_gcs.go
  42. +144 −0 pkg/asset/asset_http.go
  43. +5 −3 pkg/asset/asset_publication.go
  44. +152 −0 pkg/asset/asset_s3.go
  45. +8 −7 pkg/content/content.go
  46. +0 −3 pkg/content/element/element.go
  47. +13 −12 pkg/content/iterator/html.go
  48. +25 −23 pkg/content/iterator/html_converter.go
  49. +16 −12 pkg/content/iterator/iterator.go
  50. +24 −22 pkg/content/iterator/publication.go
  51. +9 −5 pkg/fetcher/fetcher.go
  52. +90 −27 pkg/fetcher/fetcher_archive.go
  53. +36 −34 pkg/fetcher/fetcher_archive_test.go
  54. +29 −38 pkg/fetcher/fetcher_file.go
  55. +52 −51 pkg/fetcher/fetcher_file_test.go
  56. +257 −0 pkg/fetcher/fetcher_gcs.go
  57. +257 −0 pkg/fetcher/fetcher_http.go
  58. +279 −0 pkg/fetcher/fetcher_s3.go
  59. +9 −5 pkg/fetcher/fetcher_transforming.go
  60. +176 −0 pkg/fetcher/fs.go
  61. +3 −2 pkg/fetcher/reader.go
  62. +290 −58 pkg/fetcher/resource.go
  63. +9 −24 pkg/fetcher/resource_bytes.go
  64. +17 −0 pkg/fetcher/traits.go
  65. +93 −5 pkg/manifest/a11y.go
  66. +30 −2 pkg/manifest/a11y_test.go
  67. +12 −11 pkg/manifest/collection.go
  68. +23 −22 pkg/manifest/collection_test.go
  69. +7 −7 pkg/manifest/contributor.go
  70. +4 −6 pkg/manifest/contributor_test.go
  71. +24 −0 pkg/manifest/guided_navigation.go
  72. +111 −0 pkg/manifest/href.go
  73. +62 −0 pkg/manifest/href_normalizer.go
  74. +70 −0 pkg/manifest/href_test.go
  75. +93 −119 pkg/manifest/link.go
  76. +55 −54 pkg/manifest/link_test.go
  77. +57 −11 pkg/manifest/locator.go
  78. +66 −11 pkg/manifest/locator_test.go
  79. +115 −97 pkg/manifest/manifest.go
  80. +99 −95 pkg/manifest/manifest_test.go
  81. +93 −0 pkg/manifest/manifest_transformer.go
  82. +7 −0 pkg/manifest/media_overlay.go
  83. +73 −19 pkg/manifest/metadata.go
  84. +19 −5 pkg/manifest/metadata_test.go
  85. +1 −1 pkg/manifest/profile.go
  86. +19 −0 pkg/manifest/properties.go
  87. +64 −0 pkg/manifest/properties_hash.go
  88. +7 −7 pkg/manifest/subject.go
  89. +8 −8 pkg/manifest/subject_test.go
  90. +72 −0 pkg/manifest/tdm.go
  91. +50 −0 pkg/manifest/tdm_test.go
  92. +32 −5 pkg/mediatype/mediatype.go
  93. +39 −22 pkg/mediatype/mediatype_of.go
  94. +40 −24 pkg/mediatype/sniffer.go
  95. +52 −10 pkg/mediatype/sniffer_content.go
  96. +26 −7 pkg/mediatype/sniffer_context.go
  97. +0 −10 pkg/mediatype/sniffer_mimes.go
  98. +22 −22 pkg/mediatype/sniffer_test.go
  99. +29 −24 pkg/mediatype/types.go
  100. +70 −0 pkg/mediatype/types_matcher.go
  101. +1 −3 pkg/parser/epub/clock_value.go
  102. +2 −0 pkg/parser/epub/consts.go
  103. +86 −16 pkg/parser/epub/deobfuscator.go
  104. +11 −11 pkg/parser/epub/deobfuscator_test.go
  105. +7 −6 pkg/parser/epub/factory.go
  106. +124 −0 pkg/parser/epub/media_overlay_service.go
  107. +94 −45 pkg/parser/epub/metadata.go
  108. +95 −72 pkg/parser/epub/metadata_test.go
  109. +24 −30 pkg/parser/epub/parser.go
  110. +13 −9 pkg/parser/epub/parser_encryption.go
  111. +20 −12 pkg/parser/epub/parser_encryption_test.go
  112. +11 −17 pkg/parser/epub/parser_navdoc.go
  113. +32 −30 pkg/parser/epub/parser_navdoc_test.go
  114. +19 −16 pkg/parser/epub/parser_ncx.go
  115. +28 −26 pkg/parser/epub/parser_ncx_test.go
  116. +12 −10 pkg/parser/epub/parser_packagedoc.go
  117. +43 −38 pkg/parser/epub/parser_packagedoc_test.go
  118. +141 −0 pkg/parser/epub/parser_smil.go
  119. +57 −0 pkg/parser/epub/parser_smil_test.go
  120. +25 −21 pkg/parser/epub/positions_service.go
  121. +1 −1 pkg/parser/epub/positions_service_test.go
  122. +1 −0 pkg/parser/epub/property_data_type.go
  123. +6 −0 pkg/parser/epub/testdata/package/accessibility-epub2.opf
  124. +6 −0 pkg/parser/epub/testdata/package/accessibility-epub3.opf
  125. +15 −0 pkg/parser/epub/testdata/package/tdm-epub2.opf
  126. +15 −0 pkg/parser/epub/testdata/package/tdm-epub3.opf
  127. +22 −0 pkg/parser/epub/testdata/smil/audio-clip.smil
  128. +38 −0 pkg/parser/epub/testdata/smil/audio1.smil
  129. +65 −0 pkg/parser/epub/testdata/smil/w3-10.smil
  130. +17 −0 pkg/parser/epub/testdata/smil/w3-2.smil
  131. +61 −0 pkg/parser/epub/testdata/smil/w3-3.smil
  132. +20 −0 pkg/parser/epub/testdata/smil/w3-4.smil
  133. +29 −0 pkg/parser/epub/testdata/smil/w3-8.smil
  134. +53 −7 pkg/parser/epub/utils.go
  135. +3 −1 pkg/parser/parser.go
  136. +18 −13 pkg/parser/parser_audio.go
  137. +24 −20 pkg/parser/parser_image.go
  138. +10 −7 pkg/parser/parser_image_test.go
  139. +11 −9 pkg/parser/parser_readium_webpub.go
  140. +13 −8 pkg/parser/pdf/parser.go
  141. +12 −12 pkg/parser/pdf/parser_metadata.go
  142. +15 −11 pkg/parser/pdf/positions_service.go
  143. +6 −5 pkg/parser/utils.go
  144. +9 −9 pkg/parser/utils_test.go
  145. +3 −3 pkg/{drm → protection}/drm.go
  146. +22 −34 pkg/pub/publication.go
  147. +11 −6 pkg/pub/service.go
  148. +12 −10 pkg/pub/service_content.go
  149. +74 −0 pkg/pub/service_guided_navigation.go
  150. +22 −20 pkg/pub/service_positions.go
  151. +25 −21 pkg/pub/service_positions_test.go
  152. +6 −6 pkg/streamer/a11y_infer.go
  153. +3 −3 pkg/streamer/a11y_infer_test.go
  154. +6 −5 pkg/streamer/streamer.go
  155. +0 −112 pkg/util/href.go
  156. +0 −212 pkg/util/href_test.go
  157. +0 −102 pkg/util/uri_template.go
  158. +0 −55 pkg/util/uri_template_test.go
  159. +47 −0 pkg/util/url/scheme.go
  160. +2 −0 pkg/util/url/uritemplates/README.txt
  161. +248 −0 pkg/util/url/uritemplates/uritemplates.go
  162. +280 −0 pkg/util/url/uritemplates/uritemplates_test.go
  163. +32 −0 pkg/util/url/uritemplates/utils.go
  164. +426 −0 pkg/util/url/url.go
  165. +37 −0 pkg/util/url/url_remote.go
  166. +427 −0 pkg/util/url/url_test.go
  167. +67 −0 pkg/util/version/version.go
13 changes: 0 additions & 13 deletions .eslintrc.js

This file was deleted.

24 changes: 24 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
version: 2
updates:
- package-ecosystem: github-actions
directory: /
schedule:
interval: monthly
open-pull-requests-limit: 5

- package-ecosystem: docker
directory: /
schedule:
interval: monthly
open-pull-requests-limit: 5

- package-ecosystem: gomod
directory: /
schedule:
interval: monthly
open-pull-requests-limit: 5
groups:
golang:
applies-to: security-updates
patterns:
- "golang.org*"
19 changes: 14 additions & 5 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -2,21 +2,30 @@ name: Build and Test

on:
push:
branches: [ main ]
branches: [ main, develop ]
pull_request:
branches: [ '**' ]

permissions:
contents: read

jobs:

build:
runs-on: ubuntu-latest
runs-on: [self-hosted, arm64]
steps:
- uses: actions/checkout@v2
- name: Harden Runner
uses: step-security/harden-runner@0634a2670c59f64b4a01f0f96f84700a4088b9f0 # v2.12.0
with:
egress-policy: audit

- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2

- name: Set up Go
uses: actions/setup-go@v2
uses: actions/setup-go@0aaccfd150d50ccaeb58ebd88d36e91967a5f35b # v5.4.0
with:
go-version: 1.21
go-version: '>=1.23.0'
cache: false

- name: Build
run: go build -v ./...
78 changes: 78 additions & 0 deletions .github/workflows/codeql.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# For most projects, this workflow file will not need changing; you simply need
# to commit it to your repository.
#
# You may wish to alter this file to override the set of languages analyzed,
# or to provide custom queries or build logic.
#
# ******** NOTE ********
# We have attempted to detect the languages in your repository. Please check
# the `language` matrix defined below to confirm you have the correct set of
# supported CodeQL languages.
#
name: "CodeQL"

on:
push:
branches: ["develop"]
pull_request:
# The branches below must be a subset of the branches above
branches: ["develop"]
schedule:
- cron: "0 0 * * 1"

permissions:
contents: read

jobs:
analyze:
name: Analyze
runs-on: ubuntu-latest
permissions:
actions: read
contents: read
security-events: write

strategy:
fail-fast: false
matrix:
language: ["go"]
# CodeQL supports [ $supported-codeql-languages ]
# Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support

steps:
- name: Harden Runner
uses: step-security/harden-runner@0634a2670c59f64b4a01f0f96f84700a4088b9f0 # v2.12.0
with:
egress-policy: audit

- name: Checkout repository
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2

# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@28deaeda66b76a05916b6923827895f2b14ab387 # v3.28.16
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
# By default, queries listed here will override any specified in a config file.
# Prefix the list here with "+" to use these queries and those in the config file.

# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild
uses: github/codeql-action/autobuild@28deaeda66b76a05916b6923827895f2b14ab387 # v3.28.16

# ℹ️ Command-line programs to run using the OS shell.
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun

# If the Autobuild fails above, remove it and uncomment the following three lines.
# modify them (or add more) to build your code if your project, please refer to the EXAMPLE below for guidance.

# - run: |
# echo "Run, Build Application using script"
# ./location_of_script_within_repo/buildscript.sh

- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@28deaeda66b76a05916b6923827895f2b14ab387 # v3.28.16
with:
category: "/language:${{matrix.language}}"
27 changes: 27 additions & 0 deletions .github/workflows/dependency-review.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Dependency Review Action
#
# This Action will scan dependency manifest files that change as part of a Pull Request,
# surfacing known-vulnerable versions of the packages declared or updated in the PR.
# Once installed, if the workflow run is marked as required,
# PRs introducing known-vulnerable packages will be blocked from merging.
#
# Source repository: https://github.com/actions/dependency-review-action
name: 'Dependency Review'
on: [pull_request]

permissions:
contents: read

jobs:
dependency-review:
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@0634a2670c59f64b4a01f0f96f84700a4088b9f0 # v2.12.0
with:
egress-policy: audit

- name: 'Checkout Repository'
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- name: 'Dependency Review'
uses: actions/dependency-review-action@ce3cf9537a52e8119d91fd484ab5b8a807627bf8 # v4.6.0
30 changes: 0 additions & 30 deletions .github/workflows/release.yml

This file was deleted.

76 changes: 76 additions & 0 deletions .github/workflows/scorecards.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# This workflow uses actions that are not certified by GitHub. They are provided
# by a third-party and are governed by separate terms of service, privacy
# policy, and support documentation.

name: Scorecard supply-chain security
on:
# For Branch-Protection check. Only the default branch is supported. See
# https://github.com/ossf/scorecard/blob/main/docs/checks.md#branch-protection
branch_protection_rule:
# To guarantee Maintained check is occasionally updated. See
# https://github.com/ossf/scorecard/blob/main/docs/checks.md#maintained
schedule:
- cron: '20 7 * * 2'
push:
branches: ["develop"]

# Declare default permissions as read only.
permissions: read-all

jobs:
analysis:
name: Scorecard analysis
runs-on: ubuntu-latest
permissions:
# Needed to upload the results to code-scanning dashboard.
security-events: write
# Needed to publish results and get a badge (see publish_results below).
id-token: write
contents: read
actions: read

steps:
- name: Harden Runner
uses: step-security/harden-runner@0634a2670c59f64b4a01f0f96f84700a4088b9f0 # v2.12.0
with:
egress-policy: audit

- name: "Checkout code"
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false

- name: "Run analysis"
uses: ossf/scorecard-action@f49aabe0b5af0936a0987cfb85d86b75731b0186 # v2.4.1
with:
results_file: results.sarif
results_format: sarif
# (Optional) "write" PAT token. Uncomment the `repo_token` line below if:
# - you want to enable the Branch-Protection check on a *public* repository, or
# - you are installing Scorecards on a *private* repository
# To create the PAT, follow the steps in https://github.com/ossf/scorecard-action#authentication-with-pat.
# repo_token: ${{ secrets.SCORECARD_TOKEN }}

# Public repositories:
# - Publish results to OpenSSF REST API for easy access by consumers
# - Allows the repository to include the Scorecard badge.
# - See https://github.com/ossf/scorecard-action#publishing-results.
# For private repositories:
# - `publish_results` will always be set to `false`, regardless
# of the value entered here.
publish_results: true

# Upload the results as artifacts (optional). Commenting out will disable uploads of run results in SARIF
# format to the repository Actions tab.
- name: "Upload artifact"
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: SARIF file
path: results.sarif
retention-days: 5

# Upload the results to GitHub's code scanning dashboard.
- name: "Upload to code-scanning"
uses: github/codeql-action/upload-sarif@28deaeda66b76a05916b6923827895f2b14ab387 # v3.28.16
with:
sarif_file: results.sarif
3 changes: 1 addition & 2 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,2 @@
publications/*
*.old
dist/
*.old
39 changes: 0 additions & 39 deletions .goreleaser.yaml

This file was deleted.

14 changes: 14 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
repos:
- repo: https://github.com/gitleaks/gitleaks
rev: v8.16.3
hooks:
- id: gitleaks
- repo: https://github.com/golangci/golangci-lint
rev: v1.52.2
hooks:
- id: golangci-lint
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
hooks:
- id: end-of-file-fixer
- id: trailing-whitespace
Loading