Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: tendermint/tm-db
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: master
Choose a base ref
...
head repository: tendermint/tm-db
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: marko/compaction
Choose a head ref
Can’t automatically merge. Don’t worry, you can still create the pull request.

Commits on Aug 17, 2022

  1. remove badger & bolt (#1)

    tac0turtle authored Aug 17, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    f6a5509 View commit details
  2. rename go mod (#2)

    * rename go mod
    
    * rename go mod
    tac0turtle authored Aug 17, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    8b16994 View commit details
  3. Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    e956777 View commit details
  4. remove ioutil (#15)

    faddat authored Aug 17, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    bb48343 View commit details

Commits on Aug 18, 2022

  1. Update settings.json (#20)

    faddat authored Aug 18, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    6dd980c View commit details
  2. Update ci.yml (#18)

    faddat authored Aug 18, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    76268ff View commit details
  3. First: change golangci-lint settings for atomicity (#17)

    * change golangci-lint settings for atomicity
    
    * all lints fixed here
    
    * gofumpt
    
    * change errcheck in backend_test.go
    
    * Update cleveldb_test.go
    
    * gofumpt
    faddat authored Aug 18, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    6bf6a86 View commit details
  4. Update dependabot.yml (#19)

    faddat authored Aug 18, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    285f363 View commit details
  5. Create codeql-analysis.yml (#12)

    * Create codeql-analysis.yml
    
    * Update codeql-analysis.yml
    
    Co-authored-by: Marko <marbar3778@yahoo.com>
    faddat and tac0turtle authored Aug 18, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    1c9ceed View commit details
  6. don't retract (#29)

    we no longer need to retract a release
    faddat authored Aug 18, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    6dc2c1b View commit details
  7. Delete codecov.yml (#27)

    we can and should replace this with a different tool
    
    Co-authored-by: Marko <marbar3778@yahoo.com>
    faddat and tac0turtle authored Aug 18, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    293e074 View commit details
  8. Delete CONTRIBUTING.md (#25)

    faddat authored Aug 18, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    bd932e9 View commit details
  9. Delete .mergify.yml (#26)

    this repository really should not be complex enough to require using mergify
    
    Co-authored-by: Marko <marbar3778@yahoo.com>
    faddat and tac0turtle authored Aug 18, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    afa439a View commit details
  10. tidy (#21)

    faddat authored Aug 18, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    2b1c796 View commit details
  11. move releasing info (#32)

    Co-authored-by: Marko <marbar3778@yahoo.com>
    faddat and tac0turtle authored Aug 18, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    2335788 View commit details
  12. Update .gitignore (#24)

    Co-authored-by: Marko <marbar3778@yahoo.com>
    faddat and tac0turtle authored Aug 18, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    44dbcda View commit details
  13. new dockerfile (#23)

    * new dockerfile and rename Makefile
    
    * add new Dockerfile
    
    * Update settings.json
    
    * Update settings.json
    faddat authored Aug 18, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    942e0be View commit details
  14. Update Dockerfile (#33)

    faddat authored Aug 18, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    f0ed4b3 View commit details
  15. Delete stale.yml (#36)

    I think we can easily close stale issues/PRs
    faddat authored Aug 18, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    cad0422 View commit details
  16. Update .gitpod.yml (#34)

    * Update .gitpod.yml
    
    * Update ci.yml
    
    * Update ci.yml
    faddat authored Aug 18, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    632337c View commit details

Commits on Aug 19, 2022

  1. build(deps): bump docker/build-push-action from 2 to 3 (#37)

    Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 2 to 3.
    - [Release notes](https://github.com/docker/build-push-action/releases)
    - [Commits](docker/build-push-action@v2...v3)
    
    ---
    updated-dependencies:
    - dependency-name: docker/build-push-action
      dependency-type: direct:production
      update-type: version-update:semver-major
    ...
    
    Signed-off-by: dependabot[bot] <support@github.com>
    
    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    dependabot[bot] authored Aug 19, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    f99a2d9 View commit details

Commits on Aug 21, 2022

  1. Pebble (#5)

    * vscode config to make using gitpod pleasant
    
    * pebble
    
    * Update db.go
    
    * Update Dockerfile
    
    * Update docker.yml
    
    * pebble and some fumpt
    
    * golangci-lint run ./... --fix
    
    * fumpt
    
    * gorocksdb
    
    * zero remotedb users, remove complexity
    
    * temporarily use gitpod image for faster work
    
    * dockerfile can go in the usual place
    
    * clean up golangci
    
    * temporarily use a different image so can actually run tests
    
    * rename makefile and remove how to release
    
    * Update build.yml
    
    * new build env is really light anyway
    
    * remove ioutil
    
    * fix lints
    
    * fix lints
    
    * clean Makefile
    
    * format
    
    * dependabot
    
    * errcheck cleveldb
    
    * Revert "zero remotedb users, remove complexity"
    
    This reverts commit a2f5628b689e2c2ec9b19a36cbbaeccbbe366538.
    
    * Revert "remove ioutil"
    
    This reverts commit e5330af5cf57bdbec9e2205f39f937bddf095a95.
    
    * Revert "fix lints"
    
    This reverts commit 83d7fa4d5a9fb499a45caf9f5aff86ac706a67d0.
    
    * Revert "fix lints"
    
    This reverts commit 34a4cd88b1dfd631ed9abb5e351b114f7214215a.
    
    * Update README.md
    
    * restoring remotedb
    
    * restore docs
    
    * remove last sign of badgers
    
    * upsdate db descriptions
    
    * pebble
    
    * settings fix
    
    * settings xi
    faddat authored Aug 21, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    97bc8c9 View commit details

Commits on Aug 22, 2022

  1. remove remotedb (#22)

    * removes docs folder and remotedb
    
    * Update makefile
    
    * restore docs
    
    * cleanup ci
    faddat authored Aug 22, 2022

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    23a8145 View commit details

Commits on Aug 24, 2022

  1. Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
    Copy the full SHA
    f47ae8d View commit details

Commits on Aug 25, 2022

  1. add more compaction

    tac0turtle committed Aug 25, 2022
    Copy the full SHA
    bac3b35 View commit details
Showing with 975 additions and 4,745 deletions.
  1. +1 −4 .github/dependabot.yml
  2. +0 −30 .github/workflows/ci.yml
  3. +73 −0 .github/workflows/codeql-analysis.yml
  4. +11 −41 .github/workflows/docker.yml
  5. +2 −6 .github/workflows/lint.yml
  6. +0 −20 .github/workflows/stale.yml
  7. +14 −0 .github/workflows/test.yml
  8. +2 −8 .gitignore
  9. +1 −2 .gitpod.yml
  10. +0 −15 .golangci.yml
  11. +0 −15 .mergify.yml
  12. +6 −0 .vscode/settings.json
  13. +1 −193 CHANGELOG.md
  14. +0 −8 CONTRIBUTING.md
  15. +5 −0 Dockerfile
  16. +41 −0 Makefile
  17. +9 −15 README.md
  18. +5 −5 docs/how_to_release.md → RELEASING.md
  19. +8 −10 backend_test.go
  20. +0 −295 badger_db.go
  21. +0 −204 boltdb.go
  22. +0 −87 boltdb_batch.go
  23. +0 −142 boltdb_iterator.go
  24. +0 −47 boltdb_test.go
  25. +6 −4 cleveldb_test.go
  26. +0 −26 codecov.yml
  27. +5 −5 common_test.go
  28. +5 −11 db.go
  29. +20 −25 go.mod
  30. +198 −91 go.sum
  31. +0 −71 makefile
  32. +480 −0 pebble.go
  33. +50 −0 pebble_test.go
  34. +1 −1 prefixdb_test.go
  35. +0 −84 remotedb/batch.go
  36. +0 −37 remotedb/doc.go
  37. +0 −22 remotedb/grpcdb/client.go
  38. +0 −32 remotedb/grpcdb/doc.go
  39. +0 −52 remotedb/grpcdb/example_test.go
  40. +0 −244 remotedb/grpcdb/server.go
  41. +0 −138 remotedb/iterator.go
  42. +0 −1,616 remotedb/proto/defs.pb.go
  43. +0 −78 remotedb/proto/defs.proto
  44. +0 −640 remotedb/proto/defspb_test.go
  45. +0 −123 remotedb/remotedb.go
  46. +0 −167 remotedb/remotedb_test.go
  47. +0 −25 remotedb/test.crt
  48. +0 −27 remotedb/test.key
  49. +15 −15 rocksdb.go
  50. +3 −3 rocksdb_batch.go
  51. +4 −4 rocksdb_iterator.go
  52. +1 −1 test_helpers.go
  53. +0 −44 tools/Dockerfile
  54. +8 −7 types.go
  55. +0 −5 util_test.go
5 changes: 1 addition & 4 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -1,16 +1,13 @@

version: 2
updates:
- package-ecosystem: github-actions
directory: "/"
schedule:
interval: daily
time: "11:00"
open-pull-requests-limit: 10
- package-ecosystem: gomod
directory: "/"
schedule:
interval: daily
time: "11:00"
open-pull-requests-limit: 10
labels:
- T:dependencies
30 changes: 0 additions & 30 deletions .github/workflows/ci.yml

This file was deleted.

73 changes: 73 additions & 0 deletions .github/workflows/codeql-analysis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# For most projects, this workflow file will not need changing; you simply need
# to commit it to your repository.
#
# You may wish to alter this file to override the set of languages analyzed,
# or to provide custom queries or build logic.
#
# ******** NOTE ********
# We have attempted to detect the languages in your repository. Please check
# the `language` matrix defined below to confirm you have the correct set of
# supported CodeQL languages.
#
name: "CodeQL"

on:
push:
branches: [ "main" ]
pull_request:
# The branches below must be a subset of the branches above
branches: [ "main" ]
schedule:
- cron: '26 1 * * 5'

jobs:
analyze:
name: Analyze
runs-on: ubuntu-latest
container: ghcr.io/cosmos/cosmos-db/build-test:latest
permissions:
actions: read
contents: read
security-events: write

strategy:
fail-fast: false
matrix:
language: [ 'go' ]
# CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ]
# Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support

steps:
- name: Checkout repository
uses: actions/checkout@v3

# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v2
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
# By default, queries listed here will override any specified in a config file.
# Prefix the list here with "+" to use these queries and those in the config file.

# Details on CodeQL's query packs refer to : https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
queries: security-extended,security-and-quality


# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild
uses: github/codeql-action/autobuild@v2

# ℹ️ Command-line programs to run using the OS shell.
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun

# If the Autobuild fails above, remove it and uncomment the following three lines.
# modify them (or add more) to build your code if your project, please refer to the EXAMPLE below for guidance.

# - run: |
# echo "Run, Build Application using script"
# ./location_of_script_within_repo/buildscript.sh

- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v2
52 changes: 11 additions & 41 deletions .github/workflows/docker.yml
Original file line number Diff line number Diff line change
@@ -1,56 +1,26 @@
# This workflow builds and pushes a new version of the build container image
# when the tools directory changes on master. Edit tools/Dockerfile.
#
# This workflow does not push a new image until it is merged, so tests that
# depend on changes in this image will not pass until this workflow succeeds.
# For that reason, changes here should be done in a separate PR in advance of
# work that depends on them.
name: build docker image

name: Build & Push TM-DB-Testing
on:
pull_request:
paths:
- "tools/*"
workflow_dispatch:
push:
branches:
- master
paths:
- "tools/*"
pull_request:

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Prepare
id: prep
run: |
DOCKER_IMAGE=tendermintdev/docker-tm-db-testing
VERSION=noop
if [[ $GITHUB_REF == refs/tags/* ]]; then
VERSION=${GITHUB_REF#refs/tags/}
elif [[ $GITHUB_REF == refs/heads/* ]]; then
VERSION=$(echo ${GITHUB_REF#refs/heads/} | sed -r 's#/+#-#g')
if [ "${{ github.event.repository.default_branch }}" = "$VERSION" ]; then
VERSION=latest
fi
fi
TAGS="${DOCKER_IMAGE}:${VERSION}"
echo ::set-output name=tags::${TAGS}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Checkout
uses: actions/checkout@v3

- name: Login to DockerHub
- name: Login to GitHub Container Registry
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Publish to Docker Hub
- name: Build and push
uses: docker/build-push-action@v3
with:
context: ./tools
file: ./tools/Dockerfile
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.prep.outputs.tags }}
tags: ghcr.io/cosmos/cosmos-db/build-test:latest
8 changes: 2 additions & 6 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
@@ -7,13 +7,9 @@ on:

jobs:
golangci:
# We need to run the linter on the same image we use for building, since it
# needs the C libraries installed for the dependencies to typecheck.
runs-on: ubuntu-latest
container: tendermintdev/docker-tm-db-testing
container: ghcr.io/cosmos/cosmos-db/build-test:latest
steps:
- uses: actions/checkout@v3
- uses: golangci/golangci-lint-action@v3.2.0
with:
args: --timeout 10m
github-token: ${{ secrets.github_token }}

20 changes: 0 additions & 20 deletions .github/workflows/stale.yml

This file was deleted.

14 changes: 14 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
name: Test
on:
push:
branches:
- master
pull_request:
jobs:
Test:
runs-on: ubuntu-latest
container: ghcr.io/cosmos/cosmos-db/build-test:latest
steps:
- uses: actions/checkout@v3
- name: run tests
run: go test ./... -mod=readonly -timeout 8m -race -coverprofile=coverage.txt -covermode=atomic -tags=memdb,goleveldb,cleveldb,rocksdb -v
10 changes: 2 additions & 8 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,16 +1,10 @@
# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib

# Test binary, build with `go test -c`
*.test

# Output of the go coverage tool, specifically when used with LiteIDE
*.out

.idea
.vscode
.vscode/*
.vscode/settings.json
vendor/*
3 changes: 1 addition & 2 deletions .gitpod.yml
Original file line number Diff line number Diff line change
@@ -2,6 +2,5 @@
# Please adjust to your needs (see https://www.gitpod.io/docs/config-gitpod-file)
# and commit this file to your remote git repository to share the goodness with others.

image: tendermintdev/docker-tm-db-testing

image: ghcr.io/cosmos/cosmos-db/build-test:latest
# this means that there's a one-click known good environment available to developers.
15 changes: 0 additions & 15 deletions .golangci.yml
Original file line number Diff line number Diff line change
@@ -32,18 +32,3 @@ run:
build-tags:
- cleveldb
- rocksdb
- badgerdb
- boltdb

issues:
exclude-rules:
- path: _test\.go
linters:
- gosec
linters-settings:
maligned:
suggest-new: true
errcheck:
# report about assignment of errors to blank identifier: `num, _ := strconv.Atoi(numStr)`;
# default is false: such cases aren't reported by default.
check-blank: true
15 changes: 0 additions & 15 deletions .mergify.yml

This file was deleted.

6 changes: 6 additions & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"go.buildTags": "cleveldb rocksdb pebbledb",
"go.lintTool": "golangci-lint",
"go.lintOnSave": "workspace",
"go.formatTool": "gofumpt"
}
194 changes: 1 addition & 193 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -2,196 +2,4 @@

## Unreleased

- remove mutex from prefixdb

## 0.6.7

**2022-2-21**

- Use cosmos/rocksdb instead of techbot/rocksdb
- Add `ForceCopmact` to goleveldb database

## 0.6.6

**2021-11-08**

**Important note:** Version v0.6.5 was accidentally tagged and should be
avoided. This version is identical to v0.6.4 in package structure and API, but
has updated the version marker so that normal `go get` upgrades will not
require modifying existing use of v0.6.4.

### Version bumps (since v0.6.4)

- Bump grpc from to 1.42.0.
- Bump dgraph/badger to v2 2.2007.3.
- Bump go.etcd.io/bbolt to 1.3.6.

## 0.6.5

**2021-08-04**

**Important note**: This version was tagged by accident, and should not be
used. The tag now points to the [package-reorg
branch](https://github.com/tendermint/tm-db/tree/package-reorg) so that
any existing dependencies should not break.

## 0.6.4

**2021-02-09**

Bump protobuf to 1.3.2 and grpc to 1.35.0.

## 0.6.3

**2020-11-10**

### Improvements

- [goleveldb] [\#134](https://github.com/tendermint/tm-db/pull/134) Improve iterator performance by bounding underlying iterator range (@klim0v)

## 0.6.2

**2020-08-27**

Bump grpc, badger and goleveldb (see versions in go.mod file)

## 0.6.1

**2020-08-12**

### Improvements

- [\#115](https://github.com/tendermint/tm-db/pull/115) Add a `BadgerDB` backend with build tag `badgerdb` (@mvdan)

## 0.6.0

**2020-06-24**

### Breaking Changes

- [\#99](https://github.com/tendermint/tm-db/pull/99) Keys can no longer be `nil` or empty, and values can no longer be `nil` (@erikgrinaker)

- [\#98](https://github.com/tendermint/tm-db/pull/98) `NewDB` now returns an error instead of panicing (@erikgrinaker)

- [\#96](https://github.com/tendermint/tm-db/pull/96) `Batch.Set()`, `Delete()`, and `Close()` may now error (@erikgrinaker)

- [\#97](https://github.com/tendermint/tm-db/pull/97) `Iterator.Close()` may now error (@erikgrinaker)

- [\#97](https://github.com/tendermint/tm-db/pull/97) Many iterator panics are now exposed via `Error()` instead (@erikgrinaker)

- [\#96](https://github.com/tendermint/tm-db/pull/96) The `SetDeleter` interface has been removed (@erikgrinaker)

### Bug Fixes

- [\#97](https://github.com/tendermint/tm-db/pull/97) `RemoteDB` iterators are now correctly primed with the first item when created, without calling `Next()` (@erikgrinaker)

## 0.5.2

**2020-11-10**

### Improvements

- [goleveldb] [\#134](https://github.com/tendermint/tm-db/pull/134) Improve iterator performance by bounding underlying iterator range (@klim0v)

## 0.5.1

**2020-03-30**

### Bug Fixes

- [boltdb] [\#81](https://github.com/tendermint/tm-db/pull/81) Use correct import path go.etcd.io/bbolt

## 0.5.0

**2020-03-11**

### Breaking Changes

- [\#71](https://github.com/tendermint/tm-db/pull/71) Closed or written batches can no longer be reused, all non-`Close()` calls will panic

- [memdb] [\#74](https://github.com/tendermint/tm-db/pull/74) `Iterator()` and `ReverseIterator()` now take out database read locks for the duration of the iteration

- [memdb] [\#56](https://github.com/tendermint/tm-db/pull/56) Removed some exported methods that were mainly meant for internal use: `Mutex()`, `SetNoLock()`, `SetNoLockSync()`, `DeleteNoLock()`, and `DeleteNoLockSync()`

### Improvements

- [memdb] [\#53](https://github.com/tendermint/tm-db/pull/53) Use a B-tree for storage, which significantly improves range scan performance

- [memdb] [\#56](https://github.com/tendermint/tm-db/pull/56) Use an RWMutex for improved performance with highly concurrent read-heavy workloads

### Bug Fixes

- [boltdb] [\#69](https://github.com/tendermint/tm-db/pull/69) Properly handle blank keys in iterators

- [cleveldb] [\#65](https://github.com/tendermint/tm-db/pull/65) Fix handling of empty keys as iterator endpoints

## 0.4.1

**2020-2-26**

### Breaking Changes

- [fsdb] [\#43](https://github.com/tendermint/tm-db/pull/43) Remove FSDB

### Bug Fixes

- [boltdb] [\#45](https://github.com/tendermint/tm-db/pull/45) Bring BoltDB to adhere to the db interfaces

## 0.4

**2020-1-7**

### BREAKING CHANGES

- [\#30](https://github.com/tendermint/tm-db/pull/30) Interface Breaking, Interfaces return errors instead of panic:
- Changes to function signatures:
- DB interface:
- `Get([]byte) ([]byte, error)`
- `Has(key []byte) (bool, error)`
- `Set([]byte, []byte) error`
- `SetSync([]byte, []byte) error`
- `Delete([]byte) error`
- `DeleteSync([]byte) error`
- `Iterator(start, end []byte) (Iterator, error)`
- `ReverseIterator(start, end []byte) (Iterator, error)`
- `Close() error`
- `Print() error`
- Batch interface:
- `Write() error`
- `WriteSync() error`
- Iterator interface:
- `Error() error`

### IMPROVEMENTS

- [remotedb] [\#34](https://github.com/tendermint/tm-db/pull/34) Add proto file tests and regenerate remotedb.pb.go

## 0.3

**2019-11-18**

Special thanks to external contributors on this release:

### BREAKING CHANGES

- [\#26](https://github.com/tendermint/tm-db/pull/26/files) Rename `DBBackendtype` to `BackendType`

## 0.2

**2019-09-19**

Special thanks to external contributors on this release: @stumble

### Features

- [\#12](https://github.com/tendermint/tm-db/pull/12) Add `RocksDB` (@stumble)

## 0.1

**2019-07-16**

Special thanks to external contributors on this release:

### Initial Release

- `db`, `random.go`, `bytes.go` and `os.go` from the tendermint repo.
- Removed Badger & Boltdb
8 changes: 0 additions & 8 deletions CONTRIBUTING.md

This file was deleted.

5 changes: 5 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# This file makes a development and test environment that includes the latest versions of relevant databases.

FROM archlinux

RUN pacman -Syyu --noconfirm go base-devel rocksdb leveldb git
41 changes: 41 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
GOTOOLS = github.com/golangci/golangci-lint/cmd/golangci-lint
PACKAGES=$(shell go list ./...)
INCLUDE = -I=${GOPATH}/src/github.com/tendermint/tm-db -I=${GOPATH}/src -I=${GOPATH}/src/github.com/gogo/protobuf/protobuf

export GO111MODULE = on

all: lint test

### go tests
## By default this will only test memdb, goleveldb, and pebbledb, which do not require cgo
test:
@echo "--> Running go test"
@go test $(PACKAGES) -tags pebbledb -v

test-cleveldb:
@echo "--> Running go test"
@go test $(PACKAGES) -tags cleveldb -v

test-rocksdb:
@echo "--> Running go test"
@go test $(PACKAGES) -tags rocksdb -v

test-pebble:
@echo "--> Running go test"
@go test $(PACKAGES) -tags pebbledb -v


test-all:
@echo "--> Running go test"
@go test $(PACKAGES) -tags cleveldb,rocksdb,pebbledb -v

lint:
@echo "--> Running linter"
@golangci-lint run
@go mod verify
.PHONY: lint

format:
find . -name '*.go' -type f -not -path "*.git*" -not -name '*.pb.go' -not -name '*pb_test.go' | xargs gofumpt -w -l .
.PHONY: format

24 changes: 9 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
# Tendermint DB

[![version](https://img.shields.io/github/tag/tendermint/tm-db.svg)](https://github.com/tendermint/tm-db/releases/latest)
[![license](https://img.shields.io/github/license/tendermint/tm-db.svg)](https://github.com/tendermint/tm-db/blob/master/LICENSE)
[![API Reference](https://camo.githubusercontent.com/915b7be44ada53c290eb157634330494ebe3e30a/68747470733a2f2f676f646f632e6f72672f6769746875622e636f6d2f676f6c616e672f6764646f3f7374617475732e737667)](https://pkg.go.dev/github.com/tendermint/tm-db)
[![codecov](https://codecov.io/gh/tendermint/tm-db/branch/master/graph/badge.svg)](https://codecov.io/gh/tendermint/tm-db)
![Lint](https://github.com/tendermint/tm-db/workflows/Lint/badge.svg?branch=master)
![Test](https://github.com/tendermint/tm-db/workflows/Test/badge.svg?branch=master)
# Cosmos DB

[![version](https://img.shields.io/github/tag/cosmos/cosmos-db.svg)](https://github.com/cosmos/cosmos-db/releases/latest)
[![license](https://img.shields.io/github/license/cosmos/cosmos-db.svg)](https://github.com/cosmos/cosmos-db/blob/master/LICENSE)
[![API Reference](https://camo.githubusercontent.com/915b7be44ada53c290eb157634330494ebe3e30a/68747470733a2f2f676f646f632e6f72672f6769746875622e636f6d2f676f6c616e672f6764646f3f7374617475732e737667)](https://pkg.go.dev/github.com/cosmos/cosmos-db)
[![codecov](https://codecov.io/gh/cosmos/cosmos-db/branch/master/graph/badge.svg)](https://codecov.io/gh/cosmos/cosmos-db)
![Lint](https://github.com/cosmos/cosmos-db/workflows/Lint/badge.svg?branch=master)
![Test](https://github.com/cosmos/cosmos-db/workflows/Test/badge.svg?branch=master)
[![Discord chat](https://img.shields.io/discord/669268347736686612.svg)](https://discord.gg/AzefAFd)

Common database interface for various database backends. Primarily meant for applications built on [Tendermint](https://github.com/tendermint/tendermint), such as the [Cosmos SDK](https://github.com/cosmos/cosmos-sdk), but can be used independently of these as well.

### Minimum Go Version

Go 1.13+
Go 1.19+

## Supported Database Backends

@@ -22,18 +22,12 @@ Go 1.13+

- **[LevelDB](https://github.com/google/leveldb) [experimental]:** A [Go wrapper](https://github.com/jmhodges/levigo) around [LevelDB](https://github.com/google/leveldb). Uses LSM-trees for on-disk storage, which have good performance for write-heavy workloads, particularly on spinning disks, but requires periodic compaction to maintain decent read performance and reclaim disk space. Does not support transactions.

- **[BoltDB](https://github.com/etcd-io/bbolt) [experimental]:** A [fork](https://github.com/etcd-io/bbolt) of [BoltDB](https://github.com/boltdb/bolt). Uses B+trees for on-disk storage, which have good performance for read-heavy workloads and range scans. Supports serializable ACID transactions.

- **[RocksDB](https://github.com/cosmos/gorocksdb) [experimental]:** A [Go wrapper](https://github.com/cosmos/gorocksdb) around [RocksDB](https://rocksdb.org). Similarly to LevelDB (above) it uses LSM-trees for on-disk storage, but is optimized for fast storage media such as SSDs and memory. Supports atomic transactions, but not full ACID transactions.

- **[BadgerDB](https://github.com/dgraph-io/badger) [experimental]:** A key-value database written as a pure-Go alternative to e.g. LevelDB and RocksDB, with LSM-tree storage. Makes use of multiple goroutines for performance, and includes advanced features such as serializable ACID transactions, write batches, compression, and more.

## Meta-databases

- **PrefixDB [stable]:** A database which wraps another database and uses a static prefix for all keys. This allows multiple logical databases to be stored in a common underlying databases by using different namespaces. Used by the Cosmos SDK to give different modules their own namespaced database in a single application database.

- **RemoteDB [experimental]:** A database that connects to distributed Tendermint db instances via [gRPC](https://grpc.io/). This can help with detaching difficult deployments such as LevelDB, and can also ease dependency management for Tendermint developers.

## Tests

To test common databases, run `make test`. If all databases are available on the local machine, use `make test-all` to test them all.
10 changes: 5 additions & 5 deletions docs/how_to_release.md → RELEASING.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,22 @@
# How To Release tm db
# How To Release cosmos-db

This document provides a step-by-step guide for creating a release of tm-db.
This document provides a step-by-step guide for creating a release of cosmos-db.

## 1. Update the changelog

Open the `CHANGELOG.md` at the root of the repository.
Amend the top of this file with a section for the latest version (0.6.x etc).
Amend the top of this file with a section for the latest version (eg: v1.0.x).
Be sure to include any bug fixes, improvements, dependency upgrades, and breaking changes included in this version.
(It's OK to exclude changes to tooling dependencies, like updates to Github Actions.)
Finally, create a pull request for the changelog update.
Once the tests pass and the pull request is approved, merge the change into master.

## 2. Tag the latest commit with the latest version

tm-db is provided as a golang [module](https://blog.golang.org/publishing-go-modules), which rely on git tags for versioning information.
cosmos-db is provided as a golang [module](https://blog.golang.org/publishing-go-modules), which rely on git tags for versioning information.

Tag the changelog commit in master created in step 1 with the latest version.
Be sure to prefix the version tag with `v`. For example, `v0.6.5` for version 0.6.5.
Be sure to prefix the version tag with `v`. For example, `v1.0.0` for version 1.0.0.
This tagging can be done [using github](https://docs.github.com/en/desktop/contributing-and-collaborating-using-github-desktop/managing-commits/managing-tags#creating-a-tag) or [using git](https://git-scm.com/book/en/v2/Git-Basics-Tagging) on the command line.

Note that the golang modules tooling expects tags to be immutable.
18 changes: 8 additions & 10 deletions backend_test.go
Original file line number Diff line number Diff line change
@@ -2,7 +2,6 @@ package db

import (
"fmt"
"io/ioutil"
"os"
"path/filepath"
"testing"
@@ -13,15 +12,14 @@ import (

// Register a test backend for PrefixDB as well, with some unrelated junk data
func init() {
// nolint: errcheck
registerDBCreator("prefixdb", func(name, dir string) (DB, error) {
mdb := NewMemDB()
mdb.Set([]byte("a"), []byte{1})
mdb.Set([]byte("b"), []byte{2})
mdb.Set([]byte("t"), []byte{20})
mdb.Set([]byte("test"), []byte{0})
mdb.Set([]byte("u"), []byte{21})
mdb.Set([]byte("z"), []byte{26})
mdb.Set([]byte("a"), []byte{1}) //nolint:errcheck
mdb.Set([]byte("b"), []byte{2}) //nolint:errcheck
mdb.Set([]byte("t"), []byte{20}) //nolint:errcheck
mdb.Set([]byte("test"), []byte{0}) //nolint:errcheck
mdb.Set([]byte("u"), []byte{21}) //nolint:errcheck
mdb.Set([]byte("z"), []byte{26}) //nolint:errcheck
return NewPrefixDB(mdb, []byte("test/")), nil
}, false)
}
@@ -35,7 +33,7 @@ func cleanupDBDir(dir, name string) {

func testBackendGetSetDelete(t *testing.T, backend BackendType) {
// Default
dirname, err := ioutil.TempDir("", fmt.Sprintf("test_backend_%s_", backend))
dirname, err := os.MkdirTemp("", fmt.Sprintf("test_backend_%s_", backend))
require.Nil(t, err)
db, err := NewDB("testdb", backend, dirname)
require.NoError(t, err)
@@ -302,7 +300,7 @@ func testDBIterator(t *testing.T, backend BackendType) {
[]int64(nil), "reverse iterator from 2 (ex) to 4")

// Ensure that the iterators don't panic with an empty database.
dir2, err := ioutil.TempDir("", "tm-db-test")
dir2, err := os.MkdirTemp("", "tm-db-test")
require.NoError(t, err)
db2, err := NewDB(name, backend, dir2)
require.NoError(t, err)
295 changes: 0 additions & 295 deletions badger_db.go

This file was deleted.

204 changes: 0 additions & 204 deletions boltdb.go

This file was deleted.

87 changes: 0 additions & 87 deletions boltdb_batch.go

This file was deleted.

142 changes: 0 additions & 142 deletions boltdb_iterator.go

This file was deleted.

47 changes: 0 additions & 47 deletions boltdb_test.go

This file was deleted.

10 changes: 6 additions & 4 deletions cleveldb_test.go
Original file line number Diff line number Diff line change
@@ -25,7 +25,6 @@ func TestWithClevelDB(t *testing.T) {
t.Run("ClevelDB", func(t *testing.T) { Run(t, db) })
}

//nolint: errcheck
func BenchmarkRandomReadsWrites2(b *testing.B) {
b.StopTimer()

@@ -46,19 +45,22 @@ func BenchmarkRandomReadsWrites2(b *testing.B) {
for i := 0; i < b.N; i++ {
// Write something
{
idx := (int64(rand.Int()) % numItems)
idx := (int64(rand.Int()) % numItems) //nolint:gosec
internal[idx]++
val := internal[idx]
idxBytes := int642Bytes(idx)
valBytes := int642Bytes(val)
db.Set(
err := db.Set(
idxBytes,
valBytes,
)
if err != nil {
fmt.Println("couldn't write something: cleveldb_test.go line 53")
}
}
// Read something
{
idx := (int64(rand.Int()) % numItems)
idx := (int64(rand.Int()) % numItems) //nolint:gosec
val := internal[idx]
idxBytes := int642Bytes(idx)
valBytes, err := db.Get(idxBytes)
26 changes: 0 additions & 26 deletions codecov.yml

This file was deleted.

10 changes: 5 additions & 5 deletions common_test.go
Original file line number Diff line number Diff line change
@@ -3,8 +3,8 @@ package db
import (
"bytes"
"encoding/binary"
"io/ioutil"
"math/rand"
"os"
"testing"

"github.com/stretchr/testify/assert"
@@ -67,7 +67,7 @@ func checkValuePanics(t *testing.T, itr Iterator) {
}

func newTempDB(t *testing.T, backend BackendType) (db DB, dbDir string) {
dirname, err := ioutil.TempDir("", "db_common_test")
dirname, err := os.MkdirTemp("", "db_common_test")
require.NoError(t, err)
db, err = NewDB("testdb", backend, dirname)
require.NoError(t, err)
@@ -93,7 +93,7 @@ func benchmarkRangeScans(b *testing.B, db DB, dbSize int64) {
b.StartTimer()

for i := 0; i < b.N; i++ {
start := rand.Int63n(dbSize - rangeSize)
start := rand.Int63n(dbSize - rangeSize) //nolint:gosec
end := start + rangeSize
iter, err := db.Iterator(int642Bytes(start), int642Bytes(end))
require.NoError(b, err)
@@ -122,7 +122,7 @@ func benchmarkRandomReadsWrites(b *testing.B, db DB) {
for i := 0; i < b.N; i++ {
// Write something
{
idx := rand.Int63n(numItems)
idx := rand.Int63n(numItems) //nolint:gosec
internal[idx]++
val := internal[idx]
idxBytes := int642Bytes(idx)
@@ -136,7 +136,7 @@ func benchmarkRandomReadsWrites(b *testing.B, db DB) {

// Read something
{
idx := rand.Int63n(numItems)
idx := rand.Int63n(numItems) //nolint:gosec
valExp := internal[idx]
idxBytes := int642Bytes(idx)
valBytes, err := db.Get(idxBytes)
16 changes: 5 additions & 11 deletions db.go
Original file line number Diff line number Diff line change
@@ -15,26 +15,20 @@ const (
// - stable
GoLevelDBBackend BackendType = "goleveldb"
// CLevelDBBackend represents cleveldb (uses levigo wrapper)
// - fast
// - requires gcc
// - use cleveldb build tag (go build -tags cleveldb)
CLevelDBBackend BackendType = "cleveldb"
// MemDBBackend represents in-memory key value store, which is mostly used
// for testing.
MemDBBackend BackendType = "memdb"
// BoltDBBackend represents bolt (uses etcd's fork of bolt -
// github.com/etcd-io/bbolt)
// - EXPERIMENTAL
// - may be faster is some use-cases (random reads - indexer)
// - use boltdb build tag (go build -tags boltdb)
BoltDBBackend BackendType = "boltdb"
// RocksDBBackend represents rocksdb (uses github.com/cosmos/gorocksdb)
// - EXPERIMENTAL
// RocksDBBackend represents rocksdb (uses github.com/linxGnu/grocksdb)
// - requires gcc
// - use rocksdb build tag (go build -tags rocksdb)
RocksDBBackend BackendType = "rocksdb"

BadgerDBBackend BackendType = "badgerdb"
// PebbleDBDBBackend represents pebble (uses github.com/cockroachdb/pebble)
// - pure go
// - use pebble build tag (go build -tags pebbledb)
PebbleDBBackend BackendType = "pebbledb"
)

type dbCreator func(name string, dir string) (DB, error)
45 changes: 20 additions & 25 deletions go.mod
Original file line number Diff line number Diff line change
@@ -1,41 +1,36 @@
module github.com/tendermint/tm-db
module github.com/cosmos/cosmos-db

go 1.17
go 1.19

require (
github.com/cosmos/gorocksdb v1.2.0
github.com/dgraph-io/badger/v3 v3.2103.2
github.com/gogo/protobuf v1.3.2
github.com/cockroachdb/pebble v0.0.0-20220817183557-09c6e030a677
github.com/google/btree v1.1.2
github.com/jmhodges/levigo v1.0.0
github.com/linxGnu/grocksdb v1.7.5
github.com/stretchr/testify v1.8.0
github.com/syndtr/goleveldb v1.0.1-0.20200815110645-5c35d600f0ca
go.etcd.io/bbolt v1.3.6
google.golang.org/grpc v1.48.0
)

require (
github.com/cespare/xxhash v1.1.0 // indirect
github.com/DataDog/zstd v1.4.5 // indirect
github.com/HdrHistogram/hdrhistogram-go v1.1.2 // indirect
github.com/cespare/xxhash/v2 v2.1.1 // indirect
github.com/cockroachdb/errors v1.8.1 // indirect
github.com/cockroachdb/logtags v0.0.0-20190617123548-eb05cc24525f // indirect
github.com/cockroachdb/redact v1.0.8 // indirect
github.com/cockroachdb/sentry-go v0.6.1-cockroachdb.2 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dgraph-io/ristretto v0.1.0 // indirect
github.com/dustin/go-humanize v1.0.0 // indirect
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b // indirect
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 // indirect
github.com/golang/protobuf v1.5.2 // indirect
github.com/golang/snappy v0.0.3 // indirect
github.com/google/flatbuffers v1.12.1 // indirect
github.com/klauspost/compress v1.12.3 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/klauspost/compress v1.15.9 // indirect
github.com/kr/pretty v0.3.0 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
go.opencensus.io v0.22.5 // indirect
golang.org/x/net v0.0.0-20201021035429-f5854403a974 // indirect
golang.org/x/sys v0.0.0-20220227234510-4e6760a101f9 // indirect
golang.org/x/text v0.3.3 // indirect
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013 // indirect
google.golang.org/protobuf v1.27.1 // indirect
github.com/rogpeppe/go-internal v1.8.1 // indirect
golang.org/x/exp v0.0.0-20220722155223-a9213eeb770e // indirect
golang.org/x/net v0.0.0-20220812174116-3211cb980234 // indirect
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c // indirect
golang.org/x/sys v0.0.0-20220817070843-5a390386f1f2 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)

// Breaking changes were released with the wrong tag (use v0.6.6 or later).
retract v0.6.5
289 changes: 198 additions & 91 deletions go.sum

Large diffs are not rendered by default.

71 changes: 0 additions & 71 deletions makefile

This file was deleted.

480 changes: 480 additions & 0 deletions pebble.go

Large diffs are not rendered by default.

50 changes: 50 additions & 0 deletions pebble_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
//go:build pebbledb

package db

import (
"fmt"
"os"
"testing"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)

func TestPebbleDBBackend(t *testing.T) {
name := fmt.Sprintf("test_%x", randStr(12))
dir := os.TempDir()
db, err := NewDB(name, PebbleDBBackend, dir)
require.NoError(t, err)
defer cleanupDBDir(dir, name)

_, ok := db.(*PebbleDB)
assert.True(t, ok)
}

// func TestPebbleDBStats(t *testing.T) {
// name := fmt.Sprintf("test_%x", randStr(12))
// dir := os.TempDir()
// db, err := NewDB(name, PebbleDBBackend, dir)
// require.NoError(t, err)
// defer cleanupDBDir(dir, name)

// assert.NotEmpty(t, db.Stats())
// }

func BenchmarkPebbleDBRandomReadsWrites(b *testing.B) {
name := fmt.Sprintf("test_%x", randStr(12))
dir := os.TempDir()
db, err := NewDB(name, PebbleDBBackend, dir)
if err != nil {
b.Fatal(err)
}
defer func() {
db.Close()
cleanupDBDir("", name)
}()

benchmarkRandomReadsWrites(b, db)
}

// TODO: Add tests for pebble
2 changes: 1 addition & 1 deletion prefixdb_test.go
Original file line number Diff line number Diff line change
@@ -32,7 +32,7 @@ func taskKey(i, k int) []byte {

func randomValue() []byte {
b := make([]byte, 16)
rand.Read(b)
rand.Read(b) //nolint:gosec
return b
}

84 changes: 0 additions & 84 deletions remotedb/batch.go

This file was deleted.

37 changes: 0 additions & 37 deletions remotedb/doc.go

This file was deleted.

22 changes: 0 additions & 22 deletions remotedb/grpcdb/client.go

This file was deleted.

32 changes: 0 additions & 32 deletions remotedb/grpcdb/doc.go

This file was deleted.

52 changes: 0 additions & 52 deletions remotedb/grpcdb/example_test.go

This file was deleted.

244 changes: 0 additions & 244 deletions remotedb/grpcdb/server.go

This file was deleted.

138 changes: 0 additions & 138 deletions remotedb/iterator.go

This file was deleted.

1,616 changes: 0 additions & 1,616 deletions remotedb/proto/defs.pb.go

This file was deleted.

78 changes: 0 additions & 78 deletions remotedb/proto/defs.proto

This file was deleted.

640 changes: 0 additions & 640 deletions remotedb/proto/defspb_test.go

This file was deleted.

123 changes: 0 additions & 123 deletions remotedb/remotedb.go

This file was deleted.

167 changes: 0 additions & 167 deletions remotedb/remotedb_test.go

This file was deleted.

25 changes: 0 additions & 25 deletions remotedb/test.crt

This file was deleted.

27 changes: 0 additions & 27 deletions remotedb/test.key

This file was deleted.

30 changes: 15 additions & 15 deletions rocksdb.go
Original file line number Diff line number Diff line change
@@ -8,7 +8,7 @@ import (
"path/filepath"
"runtime"

"github.com/cosmos/gorocksdb"
"github.com/linxGnu/grocksdb"
)

func init() {
@@ -20,10 +20,10 @@ func init() {

// RocksDB is a RocksDB backend.
type RocksDB struct {
db *gorocksdb.DB
ro *gorocksdb.ReadOptions
wo *gorocksdb.WriteOptions
woSync *gorocksdb.WriteOptions
db *grocksdb.DB
ro *grocksdb.ReadOptions
wo *grocksdb.WriteOptions
woSync *grocksdb.WriteOptions
}

var _ DB = (*RocksDB)(nil)
@@ -32,11 +32,11 @@ func NewRocksDB(name string, dir string) (*RocksDB, error) {
// default rocksdb option, good enough for most cases, including heavy workloads.
// 1GB table cache, 512MB write buffer(may use 50% more on heavy workloads).
// compression: snappy as default, need to -lsnappy to enable.
bbto := gorocksdb.NewDefaultBlockBasedTableOptions()
bbto.SetBlockCache(gorocksdb.NewLRUCache(1 << 30))
bbto.SetFilterPolicy(gorocksdb.NewBloomFilter(10))
bbto := grocksdb.NewDefaultBlockBasedTableOptions()
bbto.SetBlockCache(grocksdb.NewLRUCache(1 << 30))
bbto.SetFilterPolicy(grocksdb.NewBloomFilter(10))

opts := gorocksdb.NewDefaultOptions()
opts := grocksdb.NewDefaultOptions()
opts.SetBlockBasedTableFactory(bbto)
// SetMaxOpenFiles to 4096 seems to provide a reliable performance boost
opts.SetMaxOpenFiles(4096)
@@ -47,15 +47,15 @@ func NewRocksDB(name string, dir string) (*RocksDB, error) {
return NewRocksDBWithOptions(name, dir, opts)
}

func NewRocksDBWithOptions(name string, dir string, opts *gorocksdb.Options) (*RocksDB, error) {
func NewRocksDBWithOptions(name string, dir string, opts *grocksdb.Options) (*RocksDB, error) {
dbPath := filepath.Join(dir, name+".db")
db, err := gorocksdb.OpenDb(opts, dbPath)
db, err := grocksdb.OpenDb(opts, dbPath)
if err != nil {
return nil, err
}
ro := gorocksdb.NewDefaultReadOptions()
wo := gorocksdb.NewDefaultWriteOptions()
woSync := gorocksdb.NewDefaultWriteOptions()
ro := grocksdb.NewDefaultReadOptions()
wo := grocksdb.NewDefaultWriteOptions()
woSync := grocksdb.NewDefaultWriteOptions()
woSync.SetSync(true)
database := &RocksDB{
db: db,
@@ -125,7 +125,7 @@ func (db *RocksDB) DeleteSync(key []byte) error {
return db.db.Delete(db.woSync, key)
}

func (db *RocksDB) DB() *gorocksdb.DB {
func (db *RocksDB) DB() *grocksdb.DB {
return db.db
}

6 changes: 3 additions & 3 deletions rocksdb_batch.go
Original file line number Diff line number Diff line change
@@ -3,19 +3,19 @@

package db

import "github.com/cosmos/gorocksdb"
import "github.com/linxGnu/grocksdb"

type rocksDBBatch struct {
db *RocksDB
batch *gorocksdb.WriteBatch
batch *grocksdb.WriteBatch
}

var _ Batch = (*rocksDBBatch)(nil)

func newRocksDBBatch(db *RocksDB) *rocksDBBatch {
return &rocksDBBatch{
db: db,
batch: gorocksdb.NewWriteBatch(),
batch: grocksdb.NewWriteBatch(),
}
}

8 changes: 4 additions & 4 deletions rocksdb_iterator.go
Original file line number Diff line number Diff line change
@@ -6,19 +6,19 @@ package db
import (
"bytes"

"github.com/cosmos/gorocksdb"
"github.com/linxGnu/grocksdb"
)

type rocksDBIterator struct {
source *gorocksdb.Iterator
source *grocksdb.Iterator
start, end []byte
isReverse bool
isInvalid bool
}

var _ Iterator = (*rocksDBIterator)(nil)

func newRocksDBIterator(source *gorocksdb.Iterator, start, end []byte, isReverse bool) *rocksDBIterator {
func newRocksDBIterator(source *grocksdb.Iterator, start, end []byte, isReverse bool) *rocksDBIterator {
if isReverse {
if end == nil {
source.SeekToLast()
@@ -135,7 +135,7 @@ func (itr *rocksDBIterator) assertIsValid() {
// moveSliceToBytes will free the slice and copy out a go []byte
// This function can be applied on *Slice returned from Key() and Value()
// of an Iterator, because they are marked as freed.
func moveSliceToBytes(s *gorocksdb.Slice) []byte {
func moveSliceToBytes(s *grocksdb.Slice) []byte {
defer s.Free()
if !s.Exists() {
return nil
2 changes: 1 addition & 1 deletion test_helpers.go
Original file line number Diff line number Diff line change
@@ -16,7 +16,7 @@ func randStr(length int) string {
chars := []byte{}
MAIN_LOOP:
for {
val := rand.Int63() // nolint:gosec // G404: Use of weak random number generator
val := rand.Int63() //nolint:gosec
for i := 0; i < 10; i++ {
v := int(val & 0x3f) // rightmost 6 bits
if v >= 62 { // only 62 characters in strChars
44 changes: 0 additions & 44 deletions tools/Dockerfile

This file was deleted.

15 changes: 8 additions & 7 deletions types.go
Original file line number Diff line number Diff line change
@@ -111,13 +111,14 @@ type Batch interface {
// var itr Iterator = ...
// defer itr.Close()
//
// for ; itr.Valid(); itr.Next() {
// k, v := itr.Key(); itr.Value()
// ...
// }
// if err := itr.Error(); err != nil {
// ...
// }
// for ; itr.Valid(); itr.Next() {
// k, v := itr.Key(); itr.Value()
// ...
// }
//
// if err := itr.Error(); err != nil {
// ...
// }
type Iterator interface {
// Domain returns the start (inclusive) and end (exclusive) limits of the iterator.
// CONTRACT: start, end readonly []byte
5 changes: 0 additions & 5 deletions util_test.go
Original file line number Diff line number Diff line change
@@ -25,11 +25,6 @@ func TestPrefixIteratorNoMatchNil(t *testing.T) {
// Empty iterator for db populated after iterator created.
func TestPrefixIteratorNoMatch1(t *testing.T) {
for backend := range backends {
if backend == BoltDBBackend {
t.Log("bolt does not support concurrent writes while iterating")
continue
}

t.Run(fmt.Sprintf("Prefix w/ backend %s", backend), func(t *testing.T) {
db, dir := newTempDB(t, backend)
defer os.RemoveAll(dir)